Skip to content

server : emit empty input field in anthropic streaming tool_use content_block_start#22960

Open
Biilow-Bailang wants to merge 1 commit into
ggml-org:masterfrom
Biilow-Bailang:fix/anthropic-tool-use-input-field
Open

server : emit empty input field in anthropic streaming tool_use content_block_start#22960
Biilow-Bailang wants to merge 1 commit into
ggml-org:masterfrom
Biilow-Bailang:fix/anthropic-tool-use-input-field

Conversation

@Biilow-Bailang
Copy link
Copy Markdown

Bug

The Anthropic Messages API streaming response (/v1/messages with stream: true) emits content_block_start events for tool_use blocks without the required input field. The Anthropic streaming spec requires this field to be present (as an empty object {} initially), followed by input_json_delta events that incrementally build the arguments.

Two paths emit the malformed event:

  • server_task_result_cmpl_final::to_json_anthropic_stream() at tools/server/server-task.cpp:1278
  • server_task_result_cmpl_partial::to_json_anthropic() at tools/server/server-task.cpp:1789

Adjacent thinking and text blocks at the same call sites already emit placeholder fields (thinking: "", text: ""), so the omission is a consistency oversight on the tool_use branch — same class of bug as #20090 (missing signature on thinking blocks).

Symptom

Strict Anthropic SDK parsers (e.g. Rust clients used by some agent frameworks) reject the event with:

error: failed to parse Anthropic response for model <name>:
       missing field `input`;
       body: {"type":"content_block_start","index":1,
              "content_block":{"type":"tool_use","id":"...","name":"write_file"}}

This breaks any conversation that triggers a tool call. Plain-text conversations are unaffected, which is why this hasn't been noticed earlier — most users either go through the OpenAI-compatible endpoint or use the official Anthropic Python/TS SDKs which silently default missing fields to {}.

Reproduction (against an unpatched llama-server):

curl -sN -X POST http://127.0.0.1:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "any",
    "max_tokens": 100,
    "stream": true,
    "tools": [{"name":"write_file","description":"write file",
               "input_schema":{"type":"object",
                               "properties":{"path":{"type":"string"}},
                               "required":["path"]}}],
    "tool_choice": {"type":"tool","name":"write_file"},
    "messages": [{"role":"user","content":"write a file"}]
  }' | grep content_block_start

Before this patch:

data: {"type":"content_block_start","index":0,
       "content_block":{"type":"tool_use","id":"...","name":"write_file"}}

After this patch:

data: {"type":"content_block_start","index":0,
       "content_block":{"type":"tool_use","id":"...","name":"write_file","input":{}}}

Change

Two-line additions, one in each emit site. No logic change — the input field is always emitted as an empty json::object() at content_block_start, then incrementally filled by the existing input_json_delta events.

Verified end-to-end on:

  • llama.cpp commit 78fbbc2 (master at time of PR) + the patch applied locally
  • llama-server SYCL build on Intel Arc Xe2 (Lunar Lake 226V)
  • Qwen3.5-4B Q4_0 GGUF model
  • Rust Anthropic SDK client triggering write_file tool — file is correctly produced; prior to the patch, parsing fails on the first content_block_start and the call aborts mid-stream.

Compatibility

  • The new field is what the official Anthropic streaming spec mandates, so any compliant client is either tolerant (no change) or requires it (now fixed).
  • No effect on /v1/chat/completions (OpenAI-compatible) endpoint.
  • Matches the existing pattern used for thinking ({"thinking", ""}) and text ({"text", ""}) blocks at the same call sites.

…ent_block_start

The Anthropic Messages API spec requires tool_use content blocks to include an input field, even when no arguments have been emitted yet. Two streaming paths in server-task.cpp emit tool_use content_block_start events without this field:

- to_json_anthropic_stream() (final-state path)
- to_json_anthropic() (partial/incremental path)

Strict Anthropic SDK parsers (e.g. Rust clients used by some agent frameworks) reject these events with a missing-field error, which propagates as a failed_to_parse error and breaks any conversation that triggers a tool call.

Adjacent thinking and text blocks already emit the placeholder fields (thinking: "", text: ""), so this change is consistent with the existing pattern at the same call sites.

This is the same class of bug as ggml-org#20090 (missing thinking signature) but on the tool_use path.
@Biilow-Bailang Biilow-Bailang requested a review from a team as a code owner May 12, 2026 06:57
@ggml-gh-bot
Copy link
Copy Markdown

ggml-gh-bot Bot commented May 12, 2026

Hi @Biilow-Bailang, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant