LLM does not recover after context overflow

[3inary]

Describe the bug

When the context limit is exceeded, the system does not recover gracefully.
After a context overflow error, the LLM fails to restart and the Unity Editor becomes unstable.

Steps to reproduce

1. ChatBot Demo Scene
2. Model: Llama 3.1 8B
3. Set Context Size = 512
4. Num Predict / Batch Size = 512
5. Play and enter somtihing like "Tell a absurd sifi story" and "continue" until limit is hit
6. LlamaLib error 400: request (523 tokens) exceeds the available context size (512 tokens), try increasing it
7. Stopping and starting Play mode does not recover the model
8. Editor restart required and Editor may crash on exit

LLMUnity version

3.0.1

Operating System

macOs

[3inary]

With the new v3 backend I’m seeing a context mismatch.
llm.contextSize is set to 8192 and reported correctly, but when a request fails the error shows:
request (...) exceeds the available context size (2816 tokens)

This suggests the effective runtime context is much smaller than the configured value.
I’m calling LLMAgent.Chat() and the issue occurs in a real conversation scenario, but I haven’t been able to reproduce it in a minimal test case yet.

Any hints on what could reduce the effective context size that massive (reserved tokens, backend limits, template overhead, etc.) would be appreciated.

[amakropoulos]

You probably use multiple LLMAgents connected to the same LLM.
By default, the LLM allows all the LLMAgents constructed at the beginning to operate in parallel.
It assigns a slot for each one and the context window is divided by the number of LLMAgents.

If you don't want this behaviour, you can set the Parallel Prompts of the LLM to 1.
The context window will be fully utilised but the prompt of a single LLMAgent can work at any time, and every time a new LLMAgent is used the full prompt history will have to be computed (the context cache will reset).

I'm getting a bit technical :).
Maybe I should set the Parallel Prompts to be 1 by default, I'm thinking about it.
I need to document this though, or show a warning.

[3inary]

Phew, that was unexpected simple. Thanks. :)
In my case a Parallel Prompts Tooltip like
... -1 = autodetect from clients and share Context Size
might have been enough of a hint.

[amakropoulos]

Thanks, will add it!

[amakropoulos]

Actually it seems to be a larger llama.cpp issue, there is an open issue in llama.cpp on that 😔

[hlibr]

Actually it seems to be a larger llama.cpp issue, there is an open issue in llama.cpp on that 😔

Is there a way to rollback to an older llama.cpp (e.g. b6721, mentioned in the issue as truncating correctly)?
Otherwise, the model just errors out eventually once the context gets overwhelmed. If it's not fixed in llama.cpp, or rolled to a previous version - some inbuilt context pruning will be needed.

[amakropoulos]

I'm looking into how I can patch llama.cpp up to bring back the old implementation.
Context shifting needs to happen low-level to allow also caching of the prompt otherwise the entire prompt will have to be recalculated every time.

[amakropoulos]

Rolling back (or forth) to different llama.cpp versions is always a pain because the code changes ad-hoc everywhere.

[3inary]

It's not critical with the context problem solved now.
And as far as I'm concerned, that can be sat out.