Skip to content

feat(model): add cache_control support for OpenAI and DashScope protocols#985

Open
AlbumenJ wants to merge 2 commits intoagentscope-ai:mainfrom
AlbumenJ:feat/cache-control
Open

feat(model): add cache_control support for OpenAI and DashScope protocols#985
AlbumenJ wants to merge 2 commits intoagentscope-ai:mainfrom
AlbumenJ:feat/cache-control

Conversation

@AlbumenJ
Copy link
Collaborator

Add prompt caching via cache_control: {"type": "ephemeral"} to reduce API latency and cost for both OpenAI and DashScope model integrations.

  • Add cacheControl option in GenerateOptions for model-level config
  • Add CACHE_CONTROL metadata key for per-message manual marking
  • Add cache_control field to OpenAIMessage and DashScopeMessage DTOs
  • Implement automatic strategy in formatters: apply cache_control to system messages and the last message in conversation
  • Implement metadata-based manual marking in message converters
  • Keep OpenAI and DashScope implementations fully independent
  • Add comprehensive unit tests for both protocols

🤖 Generated with [Qoder][https://qoder.com]

AgentScope-Java Version

[The version of AgentScope-Java you are working on, e.g. 1.0.9, check your pom.xml dependency version or run mvn dependency:tree | grep agentscope-parent:pom(only mac/linux)]

Description

[Please describe the background, purpose, changes made, and how to test this PR]

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has been formatted with mvn spotless:apply
  • All tests are passing (mvn test)
  • Javadoc comments are complete and follow project conventions
  • Related documentation has been updated (e.g. links, examples, etc.)
  • Code is ready for review

…cols

Add prompt caching via `cache_control: {"type": "ephemeral"}` to reduce
API latency and cost for both OpenAI and DashScope model integrations.

- Add `cacheControl` option in GenerateOptions for model-level config
- Add `CACHE_CONTROL` metadata key for per-message manual marking
- Add `cache_control` field to OpenAIMessage and DashScopeMessage DTOs
- Implement automatic strategy in formatters: apply cache_control to
  system messages and the last message in conversation
- Implement metadata-based manual marking in message converters
- Keep OpenAI and DashScope implementations fully independent
- Add comprehensive unit tests for both protocols

🤖 Generated with [Qoder][https://qoder.com]
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds prompt caching support (cache_control: {"type":"ephemeral"}) across the OpenAI-compatible and DashScope protocol implementations to reduce latency/cost, with both automatic and per-message (metadata) marking.

Changes:

  • Introduces GenerateOptions.cacheControl and MessageMetadataKeys.CACHE_CONTROL to control prompt caching globally or per-message.
  • Extends OpenAI/DashScope message DTOs and converters/formatters to emit cache_control (automatic strategy + metadata override).
  • Adds new unit tests for formatter-level cache control behavior for both OpenAI and DashScope.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
agentscope-core/src/test/java/io/agentscope/core/formatter/openai/OpenAICacheControlTest.java Adds unit tests for OpenAI cache_control auto + metadata marking behavior.
agentscope-core/src/test/java/io/agentscope/core/formatter/dashscope/DashScopeCacheControlTest.java Adds unit tests for DashScope cache_control auto + metadata marking behavior (incl. multi-agent formatter).
agentscope-core/src/main/java/io/agentscope/core/model/OpenAIChatModel.java Applies cache_control strategy at request-build time when GenerateOptions.cacheControl is enabled.
agentscope-core/src/main/java/io/agentscope/core/model/DashScopeChatModel.java Applies cache_control strategy for both chat and multi-agent DashScope formatters when enabled.
agentscope-core/src/main/java/io/agentscope/core/model/GenerateOptions.java Adds cacheControl option, Javadoc, builder support, and merge behavior.
agentscope-core/src/main/java/io/agentscope/core/message/MessageMetadataKeys.java Adds CACHE_CONTROL metadata key to manually mark messages for caching.
agentscope-core/src/main/java/io/agentscope/core/formatter/openai/dto/OpenAIMessage.java Adds cache_control field to OpenAI message DTO.
agentscope-core/src/main/java/io/agentscope/core/formatter/openai/OpenAIMessageConverter.java Applies cache_control based on Msg metadata during conversion.
agentscope-core/src/main/java/io/agentscope/core/formatter/openai/OpenAIBaseFormatter.java Implements automatic cache_control application strategy + shared constant.
agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/dto/DashScopeMessage.java Adds cache_control field to DashScope message DTO.
agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeMessageConverter.java Applies cache_control based on Msg metadata during conversion.
agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeChatFormatter.java Implements automatic cache_control application strategy + shared constant.
agentscope-core/src/main/java/io/agentscope/core/formatter/dashscope/DashScopeMultiAgentFormatter.java Implements automatic cache_control application strategy for multi-agent formatting.

…ration tests

- Replace {@code} with <code> for JSON content containing braces in
  Javadoc comments across 6 files to prevent premature tag closure
- Add cache_control integration tests to OpenAIChatModelTest verifying
  request JSON includes cache_control when option is enabled
- Add cache_control integration tests to DashScopeChatModelTest verifying
  request body via MockWebServer

🤖 Generated with [Qoder][https://qoder.com]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants