The use of a specific tool severely degrades prompt caching.

[UlookEE]

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Describe the bug
The phenomena observed so far are as follows:

When using load_artifacts_tool or preload_memory_tool, these tools generate prompts to guide the LLM.
The problem is that these tools generate dynamic content within those prompts.

load_artifacts

You have a list of artifacts: {json.dumps(artifact_names)}

preload_memory

<PAST_CONVERSATIONS>
{full_memory_text}
</PAST_CONVERSATIONS>

This is where the problem arises.

These tools use append_instructions, which appends their results to the top-level system_prompt.

Example:

system_instruction"
// omitted
You have a list of artifacts:
  ["test.txt"]

  When the user asks questions about any of the artifacts, you should call the
  `load_artifacts` function to load the artifact. Always call load_artifacts
  before answering questions related to the artifacts, regardless of whether the
  artifacts have been loaded before. Do not depend on prior answers about the
  artifacts.
  

The following content is from your previous conversations with the user.
They may be useful for answering the user's current query.
<PAST_CONVERSATIONS>
Time: 2025-10-20T21:15:13.686996
user: hi
</PAST_CONVERSATIONS>
"

This behavior corrupts all forms of prompt caching, and in the case of explicit caching, it results in severe cost inefficiencies.

As with how both static_instruction and instruction operate in the agent, these prompts should instead be appended to the bottom of the prompt.

One concern is that moving them to the bottom may cause the LLM to become confused about context. However, I believe this issue could be mitigated by making this behavior optional or by issuing a warning when prompt caching is enabled together with these tools.

To Reproduce
Using load_artifacts, preload_memory (always)

I’m ready to contribute to fixing this issue.

[UlookEE]

It seemed simple, but it seems like a significant change to the current ADK structure. Are there any patches planned?

[UlookEE]

Tool is reflected after request_processors have completed processing, so it is not possible to create dynamic instructions in Tool definition using any method with small changes.

[surajksharma07]

@UlookEE The issue occurs because load_artifacts_tool and preload_memory_tool use append_instructions() to add dynamic content (timestamps, artifact lists) to system_instruction, which invalidates Gemini's prompt cache on every request.

The fix is to append this dynamic context as user messages instead: replace llm_request.append_instructions([content]) with llm_request.contents.append(Content(role='user', parts=[Part(text=content)])) in both tools.

This keeps system_instruction static while moving dynamic data to the bottom, preserving cache hits and reducing costs significantly.
Thanks!

[UlookEE]

@surajksharma07

---

First of all, thank you for your kind reply.

You're right — using your proposed solution does make Prompt Caching work. My earlier belief that it was impossible was mistaken.
However, upon deeper analysis, the fundamental issues can be summarized as follows:

1. In base_llm_flow, _preprocess_async processes request_processors.
2. Within request_processors, contents[0] is used to create a dynamic_instruction, which is then placed at an appropriate position.
3. (In the currently proposed solution) this result is appended to the end of contents as an additional dynamic_instruction.

From my perspective, this solution has the following problems:

1. The solution lacks “coherence” (or natural consistency).

To clarify, let's define the content that should come after the cached portion (and therefore doesn’t need caching) as dynamic_instruction.

First, if we solve the issue in this way, new tool developers are likely to be confused.
At first glance, the end of contents seems like an appropriate place to insert the dynamic_instruction.
However, to verify whether that location is indeed correct, the user must analyze the internal structure of the ADK in detail.
Whether the last element of contents contains the user’s message depends on the implementation of base_llm_flow.
So, one must deeply inspect the code to know where exactly to insert their code.

In reality, the end of contents is where the user’s message is inserted.
Thus, if the tool’s instruction is appended there, the dynamic_instruction will appear after the user’s message — resulting in an incomplete LLM call.
(I’m not criticizing your explanation — I’m saying this behavior is simply confusing.)

2. There’s a high chance users will insert the prompt in an incorrect position.

As seen in issue google/adk-python#3269,
this approach increases the likelihood that users will place prompts in inappropriate locations.

3. Dynamic contents from agents and tools all get appended separately.

Since both the agent’s dynamic_instruction and each tool’s dynamic content are appended individually,
they all exist as separate “content” entries.
This can make the LLM interpret them as if multiple turns of dialogue have passed, leading to unintended behavior.

---

Root Cause Analysis

I believe the recurring issues with dynamic_instruction stem from a design flaw in the Context Caching Instruction mechanism.
The process where _InstructionsLlmRequestProcessor identifies dynamic_instruction and inserts it into contents,
and _ContentLlmRequestProcessor then rebuilds contents from it, seems more like a “makeshift workaround” than a deliberate design.

This likely arose because GenerateContentConfig includes only system_instruction but not dynamic_instruction.
To respect the design of that class, this layered logic was introduced.
It would have worked well if tool_processing didn’t exist —
but since it does, there’s now no global space for agents, tools, or later dynamic_instruction consumers to share a consistent final context.

In my view, this is a core design issue.
It would be better to fix it promptly in beginning — for example, by creating a subclass of GenerateContentConfig
that includes a dedicated space for dynamic or cached instructions.

That said, as a contributor, I may not fully understand every part of the system.
If this analysis turns out to be unnecessary, please feel free to close the issue.

Thank you.

---

[surajksharma07]

@UlookEE You're absolutely right - I reviewed the code and your architectural analysis is spot on.
The current approach with _InstructionsLlmRequestProcessor and _ContentLlmRequestProcessor using heuristic repositioning is indeed fragile, and your concern about instructions ending up after user messages is valid.
This needs a proper architectural fix with dedicated dynamic_instruction support rather than the makeshift workaround.
Let me loop in @xuanyang15 and the team to discuss the proper solution path you've outlined - this is beyond a simple fix and needs design consideration.
Thanks!

[surajksharma07]

@xuanyang15 Please have a look into it.

[xuanyang15]

@seanzhou1023 Could you please help with this issue?

[adk-bot]

This issue has been automatically marked as stale because it has not had recent activity for 7 days after a maintainer requested clarification. It will be closed if no further activity occurs within 7 days.