server : emit empty input field in anthropic streaming tool_use content_block_start#22960
Open
Biilow-Bailang wants to merge 1 commit into
Open
Conversation
…ent_block_start The Anthropic Messages API spec requires tool_use content blocks to include an input field, even when no arguments have been emitted yet. Two streaming paths in server-task.cpp emit tool_use content_block_start events without this field: - to_json_anthropic_stream() (final-state path) - to_json_anthropic() (partial/incremental path) Strict Anthropic SDK parsers (e.g. Rust clients used by some agent frameworks) reject these events with a missing-field error, which propagates as a failed_to_parse error and breaks any conversation that triggers a tool call. Adjacent thinking and text blocks already emit the placeholder fields (thinking: "", text: ""), so this change is consistent with the existing pattern at the same call sites. This is the same class of bug as ggml-org#20090 (missing thinking signature) but on the tool_use path.
|
Hi @Biilow-Bailang, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
The Anthropic Messages API streaming response (
/v1/messageswithstream: true) emitscontent_block_startevents fortool_useblocks without the requiredinputfield. The Anthropic streaming spec requires this field to be present (as an empty object{}initially), followed byinput_json_deltaevents that incrementally build the arguments.Two paths emit the malformed event:
server_task_result_cmpl_final::to_json_anthropic_stream()attools/server/server-task.cpp:1278server_task_result_cmpl_partial::to_json_anthropic()attools/server/server-task.cpp:1789Adjacent
thinkingandtextblocks at the same call sites already emit placeholder fields (thinking: "",text: ""), so the omission is a consistency oversight on thetool_usebranch — same class of bug as #20090 (missingsignatureon thinking blocks).Symptom
Strict Anthropic SDK parsers (e.g. Rust clients used by some agent frameworks) reject the event with:
This breaks any conversation that triggers a tool call. Plain-text conversations are unaffected, which is why this hasn't been noticed earlier — most users either go through the OpenAI-compatible endpoint or use the official Anthropic Python/TS SDKs which silently default missing fields to
{}.Reproduction (against an unpatched
llama-server):Before this patch:
After this patch:
Change
Two-line additions, one in each emit site. No logic change — the
inputfield is always emitted as an emptyjson::object()atcontent_block_start, then incrementally filled by the existinginput_json_deltaevents.Verified end-to-end on:
write_filetool — file is correctly produced; prior to the patch, parsing fails on the firstcontent_block_startand the call aborts mid-stream.Compatibility
/v1/chat/completions(OpenAI-compatible) endpoint.thinking({"thinking", ""}) andtext({"text", ""}) blocks at the same call sites.