Context window not reset between prompts → 400 “request exceeds available context size” with local llama-server

[girls-whocode]

Error Details

• Model: Qwen2.5-Coder-32B (LAN)
• Provider: OpenAI-compatible (local llama.cpp server)
• Status Code: 400
• Client: Continue
• Server: llama.cpp llama-server

Error Output

400 the request exceeds the available context size, try increasing it

Additional Context
Please add any additional context about the error here

Server Configuration

llama-server is started with the following flags:

/opt/llm/llama.cpp/build/bin/llama-server \
  --model /mnt/models/qwen2.5-coder-32b-instruct-q4_k_m/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf \
  --host <REDACTED> \
  --port <REDACTED> \
  --ctx-size 16384 \
  --n-predict 1024 \
  --threads 16 \
  --parallel 1 \
  --n-gpu-layers 80

Notes:

• --parallel 1 (single request at a time)
• Context size is explicitly set to 16384
• Completion cap is 1024 tokens

Observed Behavior

• Prompt 1 completes successfully.
• Prompt 2 fails immediately with a 400 context-overflow error.
• Server logs show prompt tokens continuing to increase across prompts, rather than starting fresh.

From llama.cpp logs (excerpt):

slot update_slots: new prompt, n_ctx_slot = 16384
task.n_tokens = 15953
send_error: the request exceeds the available context size

This indicates the effective prompt limit is 16384 tokens, and that Continue is sending cumulative context rather than a fresh prompt.

Once a prompt completes successfully, shouldn't the next prompt:

• Start with a fresh token budget (aside from intentional chat history)
• Not reuse or accumulate prior prompt tokens unless explicitly required

In this case, Prompt 2 is a new task and should not exceed context limits immediately.

Reproduction Steps

Prompt 1 (works)

npm run start

Output:

Error: Cannot find module '/home/jessica/repositories/python/AI-Test/backend/index.js'

(Qwen resolves issue successfully)

• Backend starts
• Frontend starts

Prompt 2 (fails)

Backend now running, and frontend started, now we need to build the frontend

Result:

400 the request exceeds the available context size

No additional files or large inputs were manually added between prompts.

Additional Notes

• This appears client-side, as llama.cpp correctly enforces the context window and reports the overflow.
• The issue reproduces consistently after several successful prompts, suggesting context accumulation across requests.
• Reducing --n-predict mitigates but does not eliminate the issue.
• The same server behaves correctly with other OpenAI-compatible clients when context is reset per request.

It appears Continue may be:

• Retaining full prior prompt context across requests, and/or
• Sending a max_tokens value that causes excessive prompt reservation

Resulting in prompt token counts exceeding the effective server context window even on short follow-up prompts.

Happy to provide additional logs or test with a debug build if helpful.

[sestinj]

Related issues:

• autoadjusting prompt tokens #6616 - Chat history accumulation causing token growth across messages (closed as stale)
• Context length calculation incorrect for Anthropic Claude models (shows 130k tokens at 70% when actually 200k) #9231 - Context length calculation incorrect for models (token counting mismatch)
• Ollama models take +50% memory when called by Continue #7583 - Ollama models taking excessive memory when called by Continue (closed as stale)
• The contextLength setting is not effective #1364 - The contextLength setting is not effective (closed)

[girls-whocode]

I did forget to mention, which I believe you realized, I am using VSCode for Linux, and the server is also Linux, which maybe obvious.

[RomneyDa]

Hi @girls-whocode, just to clarify is this two new prompts or two successive prompts?

[girls-whocode]

Two successive prompts.

[RomneyDa]

Realized my question may have sounded redundant. So 1st message -> response -> 2nd message in the same conversation? Or 1st message, new convo, 1st message?

[girls-whocode]

1st message -> response -> 2nd message in the same conversation? Yes

[RomneyDa]

Got it. The context will not reset between 2 messages in the same conversation. This is expected behavior, if the first message and its response nearly fill the context limit, and the 2nd message takes it over the limit, it will fail

[girls-whocode]

Thanks for the clarification, I understand that context is expected to accumulate across messages within the same conversation, and that exceeding the context window will result in a failure.

However, what I’m seeing here goes beyond a simple “2-message overflow” case.

This occurs after multiple successful turns (6+ in this example), even though each individual prompt is small. The issue appears to be unbounded context growth across turns, eventually exhausting the context window and KV cache, rather than a single oversized prompt.

From llama.cpp server logs (abridged):

• Turn 1:
  task.n_tokens = 7335
• Turn 2:
  task.n_tokens = 7605
• Turn 3:
  task.n_tokens = 9684
• Turn 4:
  task.n_tokens = 12026
• Turn 5:
  task.n_tokens = 18277
• Turn 6:
  task.n_tokens = 19260

By turn 6, the prompt alone exceeds 19k tokens, despite no large files or prompts being manually added between turns.

Notably:

• Each request is treated as a new prompt by the server (new prompt, n_keep = 0)

• Yet the prompt token count continues to grow monotonically

• This eventually leads to KV cache pressure and slot purging:

  decode: failed to find a memory slot for batch of size 1
  purging slot with 12486 tokens
  

This suggests the client is re-sending the full accumulated conversation history (and possibly retrieved context / tool output) on every turn, without trimming or summarization, causing inevitable overflow even with a 32k context window.

Expected behavior (from a client perspective)

• Accumulating conversational context is expected

• But the client should:

  • Trim older turns
  • Summarize prior context
  • Or otherwise bound total prompt size

Without this, longer interactive coding sessions reliably fail after several turns, even when each user message is small.

This behavior reproduces consistently with:

• --ctx-size 32768
• --parallel 1
• --n-predict 1024

And does not reproduce with other OpenAI-compatible clients against the same llama.cpp server, suggesting this is a Continue-side context management issue rather than expected server behavior.

This is using dual 3090s with NVLink and 128GB RAM, I do understand that it does not magically turn two 24 GB cards into a seamless 48 GB GPU for everything so I lowered the model from 32b to 14b and increased the ctx-size for testing. Here is the exact command I ran with the test in my previous 6 turn test.

If there is a model you recommend, although I do not see how the model would be affected by the output of the tests I performed.

/opt/llm/llama.cpp/build/bin/llama-server \
  --model /mnt/models/qwen2.5-coder-14b-instruct-q4_k_m/qwen2.5-coder-14b-instruct-q4_k_m.gguf \
  --host <REDACTED> \
  --port <REDACTED> \
  --ctx-size 32768 \           # This is the maximum total context window (prompt + generation).
  --parallel 1 \               # Limits the server to one active sequence at a time.
  --n-gpu-layers 999 \         # Put as many layers on the GPU as possible.
  --n-predict 1024 \           # Caps maximum generated tokens per request.
  --context-shift \            # Enables automatic sliding window behavior
  --threads 16                 # CPU threads

Even with 32k context and a smaller model, llama.cpp logs show task.n_tokens increasing monotonically across turns (e.g., ~7k → 19k+ by turn ~6), indicating the client is re-sending an increasingly large prompt each request. Eventually this triggers context overflow / KV cache pressure.

Other OpenAI-compatible clients against the same llama.cpp server do not show the same unbounded prompt growth under similar usage.

Happy to test any mitigation flags or provide additional logs if helpful.