Idea: Provide read your own write for the cache-backed client

[alvaroaleman]

Upstream is discussing to mark the RV offficially as comparable: kubernetes/enhancements#5505
If/When that happens, we could extend the cache-backed client to provide read your own write semantics, for example by:

• Blocking in mutating operations until the RV in the cache is >= response RV
• Blocking all reading operations in the cache until it it got to >= response RV

[sbueringer]

In general makes sense to me.

I think we should definitely have an option to turn it off though for better performance.

Blocking in mutating operations until the RV in the cache is >= response RV

I think this one absolutely makes sense

Blocking all reading operations in the cache until it it got to >= response RV

I"m not so sure about this one, I assume this would essentially block all reads for a kind if there is a single write for that kind.
I think this could slow down controllers considerably. Especially if there are a lot of reconcilers that are using the same objects.

[alvaroaleman]

Delete will be tricky due to #3059

[sbueringer]

Not sure I follow. Can't we wait until the object is gone from the cache?

(just have to be careful for the case where another object with the same name is created right after deletion)

EDIT: Ah, do you mean the case where a finalizer is set or not? I would hope we can handle both cases

1. => wait until object is gone or
2. => wait until resourceVersion is high enough

[alvaroaleman]

Yeah, because of finalizers. The problem is that since we don't deserialize the object back if it isn't gone from storage, we don't know the post-delete RV, so 2) in your list doesn't have an obvious solution

[sbueringer]

Got it. If we can't come up with a good user-facing API for Delete we could overwrite Delete in the cached client and handle the response differently there?

[serathius]

Hey everyone, I look at KEP-5505 in a little different perspective, would love to hear your opinion on this. My background is in etcd and API server cache performance/correctness, so I'm newer to the intricacies of controller-runtime.

For me, KEP-5505 is a huge deal. It addresses the complexity of eventual consistency caused by watch. A tradeoff Kubernetes made for scale that put a burden on controller authors. It unlocks the biggest advantage of etcd: the totally ordered revision for every operation. This is the same "magic" that powered the recent API server cache snapshotting improvements (https://kubernetes.io/blog/2025/09/09/kubernetes-v1-34-snapshottable-api-server-cache/,), and now it can be used to build more reliable controllers.

I've seen production incidents where this complexity leads to failure: controllers falling minutes behind, resulting in a DaemonSet creating duplicate pods or a scheduler placing pods on nodes without capacity. It's an unreasonable burden to expect every developer to perfectly handle unbounded staleness, especially when it's only discovered during an incident.

Because of this, I strongly agree with @alvaroaleman's idea of Blocking all reading operations in the cache until it got to >= response RV even make it a default guard rail. As @sbueringer mentioned, it should absolutely be disable-able for performance-critical scenarios, but the default should be safety over performance.

To address the performance concerns, perhaps the API could be more flexible than a simple on/off switch. What if a controller could specify an acceptable staleness? We could also expose informer delay as a metric (like in etcd-io/etcd#20260), allowing controllers to automatically abort reconciliation if staleness exceeds a critical threshold.

I'm not sure what the best user-facing API is or where it should live (informer, controller-runtime, etc.). That's why I'm hoping to combine my perspective with your controller expertise here. Thanks!

[sbueringer]

I think the following could also be an interesting idea:

• Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client
• If the cache catches up quickly enough (configurable "timeout"), serve from cache
• Otherwise fallback to live call

(I don't remember exactly, but I think this would be similar to how one/some of the apiserver caches are working?)

[serathius]

Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client

Right, that has been available in API as Not older than request, but not in informer (https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list).

If the cache catches up quickly enough (configurable "timeout"), serve from cache

This is what we do for consistent read in watch cache, where we wait for cache to catch up to etcd. Default timeout is 3 seconds. https://kubernetes.io/blog/2024/08/15/consistent-read-from-cache-beta/. Consistent read only differs in that instead of waiting for RV from previous write, it asks etcd what is the latest RV. Either of should be good.

Otherwise fallback to live call

I would refrain from live calls for LIST if possible. They don't have a bounded cost, even with limit as object might differ a lot in size causing high decoding cost. The only exception would be Gets on single object. API server never serves Gets from cache, only from etcd as a way to reduce lock contention. Might be something we consider to avoid issues like kubernetes/kubernetes#130767.