Thanks for stopping by to let us know something could be better!
Is your feature request related to a problem? Please describe.
I'm always frustrated when i need to send metadata about a given message (Content) or Part to the model and i can't use the metadata from Part.java because it is not sent to the model. Apparently, this metadata part is actually only supported in AI studio and it is there for the convenience of the developer but it doesn't get sent to the model so the model doesn't see it.
Describe the solution you'd like
Have metadata for both messages and parts that the model can see in both Vertex or AI Studio. the hashmap field in the genai javasdk is just perfect, but if the model can't see it, then there is no point. Normally when developers build applications like a chat bot, or an agentic workflow, they could want to use their own types for messages, parts, etc. having that metadata field there in the genai java sdk for the "benefit of the developer" is not only confusing (as every other field
Describe alternatives you've considered
1 Injecting metadata as text parts. Tried it but then Gemini starts producing metadata in its response. (like mimicing what it sees in the payload)
2 Injecting metadata as thought parts. Tried it but then Gemini doesn't see or read thoughts that are not his or user text parts with thought= true)
3 Consolidating all metadata of all messages and all parts into a single text part and put it at the end of the prompt in a message with user role (worse because then the attention heads of the LLM have to jump too much to find the metadata of a given message and fail more often) and also increases token counts because you have to put a little bit of the text from each part again so the model can link the metadata to its corresponding Content/Part lost somewhere in the middle of the prompt.
No need to add any more fields to this sdk, just make sure that the metadata hashmap gets serialized to json and sent to the model like any other field and that the backend doesn't complain (in AiStudio or in Vertex AI)
Additional context
Add any other context or screenshots about the feature request here.
Thanks for stopping by to let us know something could be better!
Is your feature request related to a problem? Please describe.
I'm always frustrated when i need to send metadata about a given message (Content) or Part to the model and i can't use the metadata from Part.java because it is not sent to the model. Apparently, this metadata part is actually only supported in AI studio and it is there for the convenience of the developer but it doesn't get sent to the model so the model doesn't see it.
Describe the solution you'd like
Have metadata for both messages and parts that the model can see in both Vertex or AI Studio. the hashmap field in the genai javasdk is just perfect, but if the model can't see it, then there is no point. Normally when developers build applications like a chat bot, or an agentic workflow, they could want to use their own types for messages, parts, etc. having that metadata field there in the genai java sdk for the "benefit of the developer" is not only confusing (as every other field
Describe alternatives you've considered
1 Injecting metadata as text parts. Tried it but then Gemini starts producing metadata in its response. (like mimicing what it sees in the payload)
2 Injecting metadata as thought parts. Tried it but then Gemini doesn't see or read thoughts that are not his or user text parts with thought= true)
3 Consolidating all metadata of all messages and all parts into a single text part and put it at the end of the prompt in a message with user role (worse because then the attention heads of the LLM have to jump too much to find the metadata of a given message and fail more often) and also increases token counts because you have to put a little bit of the text from each part again so the model can link the metadata to its corresponding Content/Part lost somewhere in the middle of the prompt.
No need to add any more fields to this sdk, just make sure that the metadata hashmap gets serialized to json and sent to the model like any other field and that the backend doesn't complain (in AiStudio or in Vertex AI)
Additional context
Add any other context or screenshots about the feature request here.