Implemented generic multimodal chat handler.

[alcoftTAO]

• Implemented a generic/global multimodal chat handler.

What does it do?

It automatically uses the model's chat template and replaces all of the model's multimodal tags with the media_marker tag.

This allows a much easier implementation for multimodal models, since the chat template doesn't need to be hard-coded for each model.

How to use it?

It is as simple as passing the clip_model_path parameter to the Llama class when created.

Note

Using the previous implementation (e.g. Qwen35ChatHandler) still works.

I'm also looking forward to implement more model architectures. Please, reply if you want me to implement any.

[JamePeng]

You can take a look at how to improve the injection process. #110

[JamePeng]

It seems there's no work on how to perform URL injection for multimedia; simply replacing it with a media marker isn't enough.

This code also needs to be removed:

if hasattr(llama, 'input_ids'):
    llama.input_ids.fill(0)

Architecture-based tag guessing should not default unknown models to Qwen-style tags. Prefer detecting media tags from the actual chat template, or better, avoid tag guessing by normalizing OpenAI content parts into placeholders before rendering.

KNOWN_MEDIA_TAGS = [
    "<|image_pad|>",
    "<|audio_pad|>",
    "<|video_pad|>",
    "<|image|>",
    "<|audio|>",
    "<|video|>",
    "[IMG]",
]

and

self._chat_format_parser_tags = [
    tag for tag in KNOWN_MEDIA_TAGS
    if tag in self.chat_format
]

In addition, a check is needed to ensure that the number of replacement markers matches the number of incoming media.

[alcoftTAO]

@JamePeng What do you think of this code?

[JamePeng]

You can test the multimodal usage of qwen3vl, qwen3.5/3.6, and gemma4.
In particular, check if the omni function of gemma4 is affected.

[JamePeng]

@alcoftTAO Hello, could you resolve some file conflicts? It seems like adding unrelated files...

[alcoftTAO]

@alcoftTAO Hello, could you resolve some file conflicts? It seems like adding unrelated files...

I'm working on this.

[alcoftTAO]

I think it's fixed. Please let me know if anything is wrong.

[alcoftTAO]

You can test the multimodal usage of qwen3vl, qwen3.5/3.6, and gemma4. In particular, check if the omni function of gemma4 is affected.

Both image and audio capabilities work as expected.

[JamePeng]

It's suggested that clip_model_path: Optional[str] = None be uniformly changed to mmproj_path: Optional[str] = None, because it's no longer just mtmd that's used for simple clipping.

It's also worth considering adding the chat_template_override: Optional[str] = None feature. I've tested it with embedding or rerank models that don't have a chat template, allowing users to easily write their own based on their token ID and pass in a temporary chat template to achieve this functionality.

Alternatively, it's unnecessary to pass in the entire metadata. You can pass in the pre-processed template_choices at the end of llama.init to speed up template retrieval, or use the chat template wrapper I added to LlamaModel last night, with name=None to retrieve the default template. Both methods can speed up and reduce unnecessary memory transfers and throughput, thus improving performance.