Input injection behaviour changed while streaming, after 0.3.32?

[asagi4]

I have some code that streams the LLM output while it generates, and injects a </think> into the prompt if it detects the model spending too much time thinking.

Previously, with 0.3.32, calling llm.create_completion with an empty prompt would just continue generation, and it was possible to straightforwardly "inject" text into the generation as the assistant by just putting it in the prompt, and otherwise just use an empty prompt to continue where it left off.

but with newer versions, something has changed and the model just completely forgets the existing context and starts producing pretty much random output as if the </think> had been a user message.

To be clear, I'm not looking for a multi-turn conversation; I only want to inject a bit of text into the assistant's current turn when it has generated too much text, to essentially force it to stop thinking.

The code in question (slightly modified to remove ComfyUI stuff):

def execute(llm, prompt, think=False, seed=-1, max_output=4096, max_think=4096, settings_json="{}"
    ):
        stop_tokens = ["<end_of_turn>", "<eos>"]
        gen_params = {"stop": stop_tokens, "max_tokens": 1024}

        try:
            settings = json.loads(settings_json)
            gen_params.update(settings)
        except ValueError as e:
            log.warning("Failed to parse settings JSON: %s. Using defaults", e)

        if "presence_penalty" in gen_params:
            gen_params["present_penalty"] = gen_params.pop("presence_penalty")
        if "repetition_penalty" in gen_params:
            gen_params["repeat_penalty"] = gen_params.pop("repetition_penalty")

        if llm.chat_handler:
            llm.chat_handler.extra_template_arguments["enable_thinking"] = think

       
        user_parts = [p for p in prompt["user"] if p["type"] != "image_url"]
        
        base_messages: list[dict] = []
        if prompt["system"]:
            base_messages.append({"role": "system", "content": prompt["system"]})
        base_messages.append({"role": "user", "content": user_parts})

        accumulated = ""
        attempt = 0
        log.info("LLM response:")
        next_prompt = ""
        force_finish = False
        while attempt < _MAX_CONTINUATIONS:
            if force_finish:
                break
            attempt += 1
            if attempt == 1:
                r = llm.create_chat_completion(
                    messages=base_messages,
                    stream=True,
                    **gen_params,
                )
            else:
                accumulated += next_prompt
                print(next_prompt, end="")
                r = llama_cpp.llama_chat_format._convert_text_completion_chunks_to_chat(
                    llm.create_completion(
                        prompt=next_prompt,
                        stream=True,
                        **gen_params,
                    )
                )
                next_prompt = ""
            for chunk in r:
                delta = chunk["choices"][0]["delta"]
                if not "content" in delta:
                    continue
                text = chunk["choices"][0]["delta"]["content"] or ""
                accumulated += text
                finish = chunk["choices"][0]["finish_reason"]
                print(text, end="")
            if think and len(accumulated) >= max_think and _THINK_CLOSE not in accumulated:
                next_prompt = " ...\nI'm overthinking, let's produce the final output.\n</think>"

            if len(accumulated) >= max_output:
                log.info("Generated too much text, aborting")
                force_finish = True
                break
            if think and _THINK_CLOSE not in accumulated:
                continue
            if finish != "length":
                print()
                break
            else:
                print()
                log.warning("Response truncated after thinking due to max tokens")
            print()

        return accumulated

Is there a better way to do this with newer versions of llama-cpp-python? It is entirely possible that this code just worked by accident previously.

[JamePeng]

You could consider using the newly added continuation prefill function after injection: https://github.com/JamePeng/llama-cpp-python?tab=readme-ov-file#continuing-assistant-responses-prefill

[asagi4]

@JamePeng I'm not sure how to use that in this case, since I'm injecting text into an ongoing turn instead of starting a new completion entirely. I guess it's unclear to me how the Llama object's state changes between completion calls. If I just inject all of the accumulated content as the assistant's response, will that actually work properly?

[asagi4]

@JamePeng I wrote some test code and don't think it works correctly by just injecting the accumulated output as the assistant response.

It appears to forcibly close the thinking part instead of continuing where the previous completion stopped, which is not the behaviour I'm looking for.

For example, if I inject a "This is a test thinking process" as an assistant role start, it renders the following:

Qwen35ChatHandler(enable_thinking=True) - Start processing
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

Qwen35ChatHandler(_init_mtmd_context): Vision support detected.
Qwen35ChatHandler(_init_mtmd_context): Audio is NOT supported by this mmproj model backend.
Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt length: 673 chars, Media count: 0.
Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt: <|im_start|>system
You are an exceptionally capable, precise, and helpful multimodal AI assistant that excels at deeply understanding and richly describing images, charts, diagrams, text in images, scenes, and any visual content,
while also answering every question accurately, clearly, and step-by-step when appropriate — always responding in the same language as the user's question, remaining polite, professional, and maximally helpful.<|im_end|>
<|im_start|>user
Please create a simple javascript canvas application that renders a rotating cube<|im_end|>
<|im_start|>assistant
<think>

</think>

This is a test thinking process<|im_end|>
<|im_start|>assistant
<think>

instead of what I would expect:

Qwen35ChatHandler(enable_thinking=True) - Start processing
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

Qwen35ChatHandler(_init_mtmd_context): Vision support detected.
Qwen35ChatHandler(_init_mtmd_context): Audio is NOT supported by this mmproj model backend.
Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt length: 673 chars, Media count: 0.
Qwen35ChatHandler(_process_mtmd_prompt): Rendered prompt: <|im_start|>system
You are an exceptionally capable, precise, and helpful multimodal AI assistant that excels at deeply understanding and richly describing images, charts, diagrams, text in images, scenes, and any visual content,
while also answering every question accurately, clearly, and step-by-step when appropriate — always responding in the same language as the user's question, remaining polite, professional, and maximally helpful.<|im_end|>
<|im_start|>user
Please create a simple javascript canvas application that renders a rotating cube<|im_end|>
<|im_start|>assistant
This is a test thinking process