Guiding and constraining VLMs

[geroldmeisinger]

There is GBNF which provides BNFs (for GGML files only):
https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md

it allows to constrain the output at certain points to certain values. like providing a json during the inference and then only allow LLM only insert the string values. does anyone know if there are similar features available for VLMs?

and Guidance: https://github.com/guidance-ai/guidance

"Start caption with" is such a feature though very simple:

        # Prepare the input text.
        caption_start = self.caption_settings['caption_start']
        if model_type in (CaptionModelType.COGAGENT, CaptionModelType.COGVLM):
            # `caption_start` is added later.
            text = prompt
        elif model_type in (CaptionModelType.LLAVA_LLAMA_3,
                            CaptionModelType.LLAVA_NEXT_34B,
                            CaptionModelType.XCOMPOSER2):
            text = prompt + caption_start
        elif prompt and caption_start:
            text = f'{prompt} {caption_start}'
        else:
            text = prompt or caption_start