Skip to content

feat: support passing images in Tool results #10410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
anakin87 merged 20 commits into from
Jan 22, 2026
Merged

feat: support passing images in Tool results #10410

anakin87 merged 20 commits into from
Jan 22, 2026

Conversation

@anakin87
Copy link
Member

@anakin87 anakin87 commented Jan 20, 2026 •
edited
Loading

Related Issues

Proposed Changes:

  • expand ToolCallResult.result to not only accepts str but also list[TextContent|ImageContent]
  • modify OpenAIResponsesChatGenerator to handle this. (Haven't implemented in OpenAIChatGenerator since the Chat Completions API only accept str in Tool results).
  • add outputs_to_result initialization parameter expand outputs_to_string parameter in Tool, to allow returning the raw Tool result (instead of stringifying it) and optionally transforming it
  • add the same parameter to ComponentTool and PipelineTool
  • refactor ToolInvoker to support this feature
  • improve serialization/deserialization of outputs_to_string
  • allow using outputs_to_string in create_tool_from_function and @tool
  • docs

How did you test it?

CI, added several tests

Notes for the reviewer

Opened this PR to get feedback. Something is still missing.

  • Tested with Anthropic: wip: Anthropic support imgs tool results haystack-core-integrations#2769. Bedrock should be similar. I am also confident that the current abstractions can work with Gemini and other providers when they support this feature.
  • Missing:
    • support new ToolCallResult in Jinja2 ChatMessage extension - not needed but in general we tried to keep feature parity between the ChatMessage dataclass and the extension - I'll open another issue for this

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@vercel
Copy link

vercel bot commented Jan 20, 2026 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
haystack-docs Ready Ready Preview, Comment Jan 22, 2026 2:59pm

Request Review

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Jan 20, 2026
@coveralls
Copy link
Collaborator

coveralls commented Jan 21, 2026 •
edited
Loading

Pull Request Test Coverage Report for Build 21252856448

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 26 unchanged lines in 6 files lost coverage.
  • Overall coverage increased (+0.1%) to 92.482%
Files with Coverage Reduction New Missed Lines %
components/generators/chat/openai_responses.py 2 87.38%
tools/pipeline_tool.py 2 93.55%
dataclasses/chat_message.py 3 99.08%
tools/component_tool.py 5 94.32%
tools/tool.py 5 96.77%
components/tools/tool_invoker.py 9 96.35%
Totals Coverage Status
Change from base Build 21252614229: 0.1%
Covered Lines: 14775
Relevant Lines: 15976
💛 - Coveralls

@anakin87 anakin87 changed the title wip: support images in tool results feat: support passing images in Tool results Jan 21, 2026
@anakin87 anakin87 marked this pull request as ready for review January 21, 2026 15:29
@anakin87 anakin87 requested a review from a team as a code owner January 21, 2026 15:29
@anakin87 anakin87 requested review from julian-risch and sjrl and removed request for a team and julian-risch January 21, 2026 15:29
sjrl
sjrl reviewed Jan 22, 2026
View reviewed changes
sjrl
sjrl reviewed Jan 22, 2026
View reviewed changes
@sjrl
Copy link
Contributor

sjrl commented Jan 22, 2026

@anakin87 if we do end up adding a new field like outputs_to_result then we should make a follow-up issue to update MCPTool and MCPToolset to support this as well.

@anakin87 anakin87 marked this pull request as draft January 22, 2026 10:36
@anakin87
Copy link
Member Author

anakin87 commented Jan 22, 2026

@sjrl It's ready for another review

@anakin87 anakin87 marked this pull request as ready for review January 22, 2026 12:14
@anakin87 anakin87 requested a review from sjrl January 22, 2026 12:14
sjrl
sjrl reviewed Jan 22, 2026
View reviewed changes
@sjrl
Copy link
Contributor

sjrl commented Jan 22, 2026

@anakin87 looking good! My only comment would be to add a release note. Thanks for adding all of the additional tests!

github-actions[bot]
github-actions bot reviewed Jan 22, 2026
View reviewed changes
docs-website/docs/tools/tool.mdx
"source": "docs", "handler": format_documents, "raw_result": False
}
```
- `source`: (Optional) Specifies the key to extract from the tool's output dictionary. If omitted, the entire result is passed to the handler.
Copy link
Contributor

@github-actions github-actions bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] reported by reviewdog 🐶
[Google.Parens] Use parentheses judiciously.

docs-website/docs/tools/tool.mdx
}
```
- `source`: (Optional) Specifies the key to extract from the tool's output dictionary. If omitted, the entire result is passed to the handler.
- `handler`: (Optional) A function that takes the output (or the extracted source value) and returns the final result.
Copy link
Contributor

@github-actions github-actions bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] reported by reviewdog 🐶
[Google.Parens] Use parentheses judiciously.

docs-website/docs/tools/tool.mdx
```
- `source`: (Optional) Specifies the key to extract from the tool's output dictionary. If omitted, the entire result is passed to the handler.
- `handler`: (Optional) A function that takes the output (or the extracted source value) and returns the final result.
- `raw_result`: (Optional) If `True`, the result is returned "as is" without further string conversion, but applying the `handler` if provided.
Copy link
Contributor

@github-actions github-actions bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] reported by reviewdog 🐶
[Google.Parens] Use parentheses judiciously.

docs-website/docs/tools/tool.mdx
}
```
Each entry defines a `source` key and can optionally include a `handler`. The individual outputs are processed,
collected into a dictionary, and then converted into a single string (usually a JSON-like representation) for the LLM.
Copy link
Contributor

@github-actions github-actions bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] reported by reviewdog 🐶
[Google.Parens] Use parentheses judiciously.

docs-website/docs/tools/tool.mdx
collected into a dictionary, and then converted into a single string (usually a JSON-like representation) for the LLM.

:::note
`raw_result` is not supported in the multiple output format.
Copy link
Contributor

@github-actions github-actions bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] reported by reviewdog 🐶
[Google.Contractions] Feel free to use 'isn't' instead of 'is not'.

Copy link
Contributor

@sjrl sjrl Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feel free to ignore this comment :D

@anakin87
Copy link
Member Author

anakin87 commented Jan 22, 2026 •
edited
Loading

@sjrl should be ready. I am trying to re-trigger docs build to see the preview.

About

support new ToolCallResult in Jinja2 ChatMessage extension - not needed but in general we tried to keep feature parity between the ChatMessage dataclass and the extension

I'll open another issue for this if you agree

@sjrl
Copy link
Contributor

sjrl commented Jan 22, 2026

@sjrl should be ready. I am trying to re-trigger docs build to see the preview.

About

support new ToolCallResult in Jinja2 ChatMessage extension - not needed but in general we tried to keep feature parity between the ChatMessage dataclass and the extension

I'll open another issue for this if you agree

Yeah I think worth opening an issue

@anakin87
Copy link
Member Author

anakin87 commented Jan 22, 2026

Docs preview: https://haystack-docs-mlymz51dw-deepset-ai.vercel.app/docs/next/tool#shaping-tool-outputs-with-outputs_to_string

sjrl
sjrl approved these changes Jan 22, 2026
View reviewed changes
Copy link
Contributor

@sjrl sjrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@anakin87 anakin87 mentioned this pull request Jan 22, 2026
Open
@anakin87
Copy link
Member Author

anakin87 commented Jan 22, 2026 •
edited
Loading

Jinja2 extension issue: #10429

@anakin87 anakin87 merged commit ecb30a7 into main Jan 22, 2026
23 checks passed
@anakin87 anakin87 deleted the support-imgs-tool-results branch January 22, 2026 15:13
This was referenced Jan 22, 2026
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@sjrl sjrl sjrl approved these changes

Assignees

No one assigned

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@anakin87 @coveralls @sjrl