feat(model): add cache_control support for OpenAI and DashScope protocols#985
Open
AlbumenJ wants to merge 2 commits intoagentscope-ai:mainfrom
Open
feat(model): add cache_control support for OpenAI and DashScope protocols#985AlbumenJ wants to merge 2 commits intoagentscope-ai:mainfrom
AlbumenJ wants to merge 2 commits intoagentscope-ai:mainfrom
Conversation
…cols
Add prompt caching via `cache_control: {"type": "ephemeral"}` to reduce
API latency and cost for both OpenAI and DashScope model integrations.
- Add `cacheControl` option in GenerateOptions for model-level config
- Add `CACHE_CONTROL` metadata key for per-message manual marking
- Add `cache_control` field to OpenAIMessage and DashScopeMessage DTOs
- Implement automatic strategy in formatters: apply cache_control to
system messages and the last message in conversation
- Implement metadata-based manual marking in message converters
- Keep OpenAI and DashScope implementations fully independent
- Add comprehensive unit tests for both protocols
🤖 Generated with [Qoder][https://qoder.com]
Contributor
There was a problem hiding this comment.
Pull request overview
Adds prompt caching support (cache_control: {"type":"ephemeral"}) across the OpenAI-compatible and DashScope protocol implementations to reduce latency/cost, with both automatic and per-message (metadata) marking.
Changes:
- Introduces
GenerateOptions.cacheControlandMessageMetadataKeys.CACHE_CONTROLto control prompt caching globally or per-message. - Extends OpenAI/DashScope message DTOs and converters/formatters to emit
cache_control(automatic strategy + metadata override). - Adds new unit tests for formatter-level cache control behavior for both OpenAI and DashScope.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| agentscope-core/src/test/java/io/agentscope/core/formatter/openai/OpenAICacheControlTest.java | Adds unit tests for OpenAI cache_control auto + metadata marking behavior. |
| agentscope-core/src/test/java/io/agentscope/core/formatter/dashscope/DashScopeCacheControlTest.java | Adds unit tests for DashScope cache_control auto + metadata marking behavior (incl. multi-agent formatter). |
| agentscope-core/src/main/java/io/agentscope/core/model/OpenAIChatModel.java | Applies cache_control strategy at request-build time when GenerateOptions.cacheControl is enabled. |
| agentscope-core/src/main/java/io/agentscope/core/model/DashScopeChatModel.java | Applies cache_control strategy for both chat and multi-agent DashScope formatters when enabled. |
| agentscope-core/src/main/java/io/agentscope/core/model/GenerateOptions.java | Adds cacheControl option, Javadoc, builder support, and merge behavior. |
| agentscope-core/src/main/java/io/agentscope/core/message/MessageMetadataKeys.java | Adds CACHE_CONTROL metadata key to manually mark messages for caching. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/openai/dto/OpenAIMessage.java | Adds cache_control field to OpenAI message DTO. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/openai/OpenAIMessageConverter.java | Applies cache_control based on Msg metadata during conversion. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/openai/OpenAIBaseFormatter.java | Implements automatic cache_control application strategy + shared constant. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/dto/DashScopeMessage.java | Adds cache_control field to DashScope message DTO. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeMessageConverter.java | Applies cache_control based on Msg metadata during conversion. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeChatFormatter.java | Implements automatic cache_control application strategy + shared constant. |
| agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeMultiAgentFormatter.java | Implements automatic cache_control application strategy for multi-agent formatting. |
agentscope-core/src/main/java/io/agentscope/core/model/OpenAIChatModel.java
Show resolved
Hide resolved
agentscope-core/src/main/java/io/agentscope/core/model/DashScopeChatModel.java
Show resolved
Hide resolved
agentscope-core/src/main/java/io/agentscope/core/model/GenerateOptions.java
Outdated
Show resolved
Hide resolved
agentscope-core/src/main/java/io/agentscope/core/model/GenerateOptions.java
Outdated
Show resolved
Hide resolved
agentscope-core/src/main/java/io/agentscope/core/message/MessageMetadataKeys.java
Outdated
Show resolved
Hide resolved
agentscope-core/src/main/java/io/agentscope/core/formatter/openai/OpenAIBaseFormatter.java
Outdated
Show resolved
Hide resolved
...tscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeChatFormatter.java
Outdated
Show resolved
Hide resolved
...-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeMultiAgentFormatter.java
Outdated
Show resolved
Hide resolved
…ration tests
- Replace {@code} with <code> for JSON content containing braces in
Javadoc comments across 6 files to prevent premature tag closure
- Add cache_control integration tests to OpenAIChatModelTest verifying
request JSON includes cache_control when option is enabled
- Add cache_control integration tests to DashScopeChatModelTest verifying
request body via MockWebServer
🤖 Generated with [Qoder][https://qoder.com]
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add prompt caching via
cache_control: {"type": "ephemeral"}to reduce API latency and cost for both OpenAI and DashScope model integrations.cacheControloption in GenerateOptions for model-level configCACHE_CONTROLmetadata key for per-message manual markingcache_controlfield to OpenAIMessage and DashScopeMessage DTOs🤖 Generated with [Qoder][https://qoder.com]
AgentScope-Java Version
[The version of AgentScope-Java you are working on, e.g. 1.0.9, check your pom.xml dependency version or run
mvn dependency:tree | grep agentscope-parent:pom(only mac/linux)]Description
[Please describe the background, purpose, changes made, and how to test this PR]
Checklist
Please check the following items before code is ready to be reviewed.
mvn spotless:applymvn test)