Skip to content

Fix inlineData and text not handled correctly for Gemini#684

Open
bubiche wants to merge 1 commit intocrmne:mainfrom
bubiche:nguyen/gemini_response_attachment
Open

Fix inlineData and text not handled correctly for Gemini#684
bubiche wants to merge 1 commit intocrmne:mainfrom
bubiche:nguyen/gemini_response_attachment

Conversation

@bubiche
Copy link
Contributor

@bubiche bubiche commented Mar 17, 2026

What this does

  • Gemini/Vertex AI can response with text and inlineData together, e.g. {"candidates": [{"content": {"role": "model", "parts": [{"text": "Here is your image! "}, {"inlineData": {"mimeType": "image/png", "data": ... }}]}. The current code doesn't handle inlineData because extract_text_parts would just get the text and be done with it.
    => Fix it by making parse_content the only code path, it'll filter out non-user-visible stuff like thought and functionCall and build content correctly.
  • I think there's also a bug inRubyLLM::Providers::Gemini::Media's build_response_content, Content.new(text:, attachments:) expects attachment sources (paths/IO/URLs), not RubyLLM::Attachment instances
    => Fix it to construct Content correctly too.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
    • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
    • All tests pass: bundle exec rspec
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant