feat(cache-provider): add on_set_prompt lifecycle hook (CORE-246)#14018
Draft
deepme987 wants to merge 1 commit into
Draft
feat(cache-provider): add on_set_prompt lifecycle hook (CORE-246)#14018deepme987 wants to merge 1 commit into
deepme987 wants to merge 1 commit into
Conversation
035c98e to
615f551
Compare
Adds a new on_set_prompt() lifecycle hook on CacheProvider that fires after the cache key set is prepared for a new prompt. Dispatched via asyncio.create_task with errors swallowed (same fail-safe pattern as on_store / on_lookup). Why: BasicCache's lifecycle notifications to external providers were incomplete. set_prompt is a key per-prompt event that providers need visibility into — for example, to reset per-prompt timing/state used for cost-aware caching policies (a provider can set t=0 here, then measure elapsed at each on_store to estimate compute saved by a hit). Backward-compatible: default implementation is a no-op, existing providers compile and run unchanged. Providers that need the per-prompt boundary override on_set_prompt().
615f551 to
fcbe7db
Compare
📝 WalkthroughWalkthroughThis PR introduces a new provider lifecycle hook to the cache system. A new 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
on_set_prompt()lifecycle hook on theCacheProviderinterface that fires after the cache key set is prepared for a new prompt. Dispatched as an async task with errors swallowed (same fail-safe pattern ason_store/on_lookup/on_prompt_start/on_prompt_end).Why
BasicCache's lifecycle notifications to external providers are incomplete.set_promptis a key per-prompt event — the moment the cache key set is prepared and the cache is ready to serve lookups — and providers currently get no signal for it. There's a clean completeness argument for surfacing this.The immediate motivating use case: cost-aware caching policies. A provider can set
t = perf_counter()inon_set_prompt, then measureelapsedat eachon_storeto estimate the compute that a future hit on this entry would save. That gives the provider a simple, accurate per-prompt clock without needing per-node timing exposed in the API.Implementation
Only 31 lines, two files, fully additive.
Backward compatibility
on_set_prompt()method has apassdefault — existing providers compile and run unchanged._notify_providers_set_promptis gated onself.enable_providers+_has_cache_providers(), so no-op when no providers are registered.Lifecycle position
Test plan
on_set_promptdefaults to no-op preserves behavior)on_set_promptand confirm it fires once per prompt, after cache keys are prepared