feat: Virtual cluster lifecycle state model

[SamBarker]

Summary

Introduces proposal 016 defining a lifecycle state model for virtual clusters. Virtual clusters are modelled as entities in the Domain-Driven Design sense — each has a persistent identity (its name) that carries through state changes. Reload is a transition the entity passes through, not a replacement of it.

The model defines five states:

• initializing — cluster being set up; not yet serving traffic
• serving — proxy has completed setup and is serving traffic
• draining — no new connections or requests accepted; existing in-flight requests given the opportunity to complete
• failed — configuration not viable; all partially-acquired resources released
• stopped — terminal; cluster permanently removed from configuration

Key Design Decisions

• Virtual clusters are entities — each cluster's identifier is its name; the state machine tracks the entity, not a configuration instance. Renaming a cluster is a destructive operation.
• serving not accepting — the state name reflects that the proxy is actively serving traffic, not merely open to new connections
• Runtime health is out of scope — serving is not split into healthy/degraded; runtime health is orthogonal to lifecycle and better addressed by readiness probes and metrics. Circuit-breaking is noted as a manifestation of the same concern.
• Draining takes the client's view of in-flight — a request is in-flight from the moment the client sends it until it receives a response. New requests on existing connections are rejected during drain to ensure the cluster quiesces.
• Graceful drain is best-effort — the drain timeout is the hard backstop; aligns with Kafka ecosystem norms (brokers drain before stopping)
• partialInitialisationPolicy: serve-none | serve-others — governs proxy behaviour when cluster initialisation fails, whether on startup or reload. Default is serve-none (matches current behaviour).
• stopped is terminal — reload routes through draining → initializing, never through stopped
• Observability is public API — management endpoint and metrics (current state, time in state, transition count) are committed to in the proposal

Relationship to Other Proposals

This provides a foundation for 012 - Hot Reload, which can define reload transitions on this state model. Note: the Hot Reload proposal's "Cluster modification: remove + add" framing describes internal runtime mechanics; from the lifecycle model's perspective the named entity persists through modification.

Test plan

• Review state definitions for clarity and completeness
• Verify state transition diagram consistency with transition descriptions
• Review rejected alternatives for completeness
• Consider whether future enhancements section adequately captures known follow-up work

[tombentley]

Thanks @SamBarker I think this is useful work, but have doubts that we really need to have "healthy" as a state.

[tombentley]

It's not immediately apparent to me how this would work in practice. Suppose I have a producer pipelining Produce requests. How do we intend to stop that flow in such a way that the producer doesn't end up with an unacknowledged request?

You mention later on about timeout, but:

• the broker doesn't know what the client's timeout it configured to, so it doesn't know how long it needs to wait
• if the client timesout request 1 while thinking that 2, 3 and 4 are in flight then it will send request 5

[SamBarker]

The exact mechanics of how the proxy tracks in-flight requests per connection are an implementation concern rather than a design one. The design's position is that draining is best-effort: clients are given a time window to complete naturally, with the drain timeout as the hard backstop after which connections are closed regardless. Kafka clients are required to handle connection loss, so forced closure after timeout is protocol-compliant — the goal is to avoid unnecessary disruption, not to guarantee zero errors. I've clarified this in the proposal.

[tombentley]

Sorry @SamBarker, but I think this does need to be explained.

In the reloading proposal it's been observed that idempotent (but not transactional) producers get their idempotency from the scope of a connection. So trying to minimise duplicate records is clearly a concern for some users, and I think it's only fair that you at least sketch how you think this will work, so everyone can see what the trade-offs are.

clients are given a time window to complete naturally

This is simply not the case, at least as far as I understand what's proposed.

For a real Kafka broker, the clean shutdown process involves the broker telling the controller that it needs to stop, the controller selects replacement leaders for all the partitions that that broker is currently leading, and only once the broker is not leading any partitions does it shut down. In that way producers avoid having any requests in-flight to the broker that's shutting down.

The proxy lacks that mechanism, which means that we are delivering a weaker idempotency guarantee under disconnections than Kafka does. That's an existing problem. But it will be made worse by building a dynamic restart feature which more frequent disconnections.

To me it looks like this is the proposal where we should be figuring out a solution and everyone can assure themselves that it will work well enough.

[SamBarker]

Sorry, I've got local changes that were meant to go out in sync with comment.

Also I thought you were pushing for the opposite solution, let the protocol handle it all and skip the drain.

I'll push the changes shortly and come back to this.

[SamBarker]

The idempotent producer concern is real and the proposal now acknowledges it explicitly. However, this is an existing proxy limitation that predates this proposal — any proxy restart today drops all connections with the same effect. A reload mechanism building on this lifecycle model actually improves the situation by confining disruption to a single virtual cluster rather than the whole process.

Solving the fundamental problem — replicating broker-style partition leader migration to avoid session disruption entirely — is a substantial piece of work well beyond the scope of a lifecycle model, with a multitude of possible solutions that the proxy cannot currently express. The proposal notes the limitation and directs users to consider how analogous situations (broker crash, network split) are handled without the proxy, since clients must already be prepared for those. Users requiring exactly-once guarantees across reconnections should use transactional producers.

[SamBarker]

The draining section has been updated to clarify in-flight semantics and correct an earlier misstatement about idempotent producers.

On the specific concern about idempotent producers: a proxy-initiated drain is no different from any other connection loss event from the client's perspective. Idempotent producers retain their Producer ID and sequence numbers in client memory across reconnections — a connection drop does not start a new session. When the client reconnects after a drain, it retries unacknowledged requests with the same (PID, epoch, sequence) tuple, and the broker deduplicates them using its existing producer state. A new PID is only assigned on producer process restart, not on reconnection. The drain timeout is the hard backstop after which connections close, but this is protocol-compliant — Kafka clients are required to handle connection loss, and the idempotent producer's deduplication mechanism is designed to survive exactly this scenario.

References:

• KIP-360 documents how the broker retains producer state and how idempotent producers can safely bump their epoch when broker state is lost — the key point being that the PID is retained client-side across reconnections.
• The Idempotent Producer design doc confirms that deduplication is based on (PID, epoch, sequence) per partition, not per connection: "The broker maintains in memory the sequence numbers it receives for each topic partition from every PID."
• KIP-854 covers broker-side producer state expiry — the state is retained well beyond any reasonable drain timeout.

The lifecycle model actually improves the posture for idempotent producers compared to the status quo. Today, any configuration change requires a full process restart, dropping every connection across every virtual cluster. With per-cluster lifecycle and reload, only the affected cluster's connections drain — the blast radius shrinks from "all clusters" to "one cluster" and planned restarts become less frequent.

I think the proposal goes far enough on draining. The lifecycle model defines that draining exists as a state and its high-level semantics (reject new connections, reject new requests, allow in-flight to complete, timeout as backstop). The detailed mechanics — specific error codes for rejected requests, interaction with pipelining, TCP-level behaviour — are implementation concerns that don't change the lifecycle state model and are better resolved in the reload proposal where draining becomes a frequent operation rather than a rare shutdown event.

[SamBarker]

@tombentley can we resolve this thread?

[tombentley]

You've defined how proxy started works wrt the VM state machines. It's not really clear to me whether we need a proposal about the rest, unless you think that that state is also exposed via metrics/mgmt etc.

[SamBarker]

Yes — proxy-level state would also be exposed via metrics and management endpoints, which is the argument for capturing it as future work. The per-cluster metrics defined in this proposal are one layer; a proxy-level lifecycle would add an aggregate layer above them covering port binding, process startup/shutdown sequencing, and overall proxy health. Defining it now felt premature, but it's a natural next step once the per-cluster model is established.

[SamBarker]

The future enhancements section flags proxy-level lifecycle as a problem space that exists, not as work we're committed to. Whether proxy-level state needs its own formal model, observable via metrics/management endpoints, or is simply emergent from the per-VC lifecycles plus standard process management is an open question — and one that might not need answering.

Every proxy has some form of process-level lifecycle (HAProxy expresses it through old/new process coexistence during reload, nginx through worker generations, httpd through child worker replacement), but none of them formalise it as a state machine — it's just how startup and shutdown work. Kroxylicious may end up the same way: the proxy starts VCs, manages their lifecycles, and exits. If that's sufficient, no proposal is needed.

I've reworded the section to make it clearer that these are genuine open questions without obvious answers (port binding significance, health aggregation across deployment models) rather than planned future work.

[SamBarker]

@tombentley similar question here, are we happy?

[tombentley]

Thanks @SamBarker, I think this is looking pretty good. I had a question is about how the configuration API for the termination behaviour.

[tombentley]

I'm on-board with the broad concept, but:

• I don't find this name to be very intuitive
• Is there an equivalent thing which defines how the proxy handles failures during reload? I don't think we'd really need a separate config for that, (or did you have a use case in mind?)

This really seems to be about specifying the user's tolerance for virtual clusters entering/remaining in the failed state. Currently the state machine diagram shows a retry loop, but the way you're describing it here is that loop is, in actual fact, the kubernetes pod crash loop. And therefore that control loop does not exist when the proxy is not running in kube.

Perhaps we should borrow some ideas from how Kubernetes allows pods to fail, but apply them directly in the proxy (whether or not it happens to be running in Kube). Specifically, some number of retries, a retry delay ± a backoff. And implicitly we terminate when we run out of retries for any VC. (You can configure the number of reties as Integer.MAX_VALUE if you don't want to terminate). We can add some other policy than terminate later on if it's needed.

[SamBarker]

I can see where your coming from. I had hoped that initialisation would be read as VC initialisation rather than proxy startup.

How about restructuring as:

proxy:
  onVirtualClusterFailure:
    serve: none          # default
    # serve: remaining   # keep serving healthy clusters

onVirtualClusterFailure avoids ambiguity with target clusters. The values describe the proxy-level consequence rather than the individual cluster's fate.

I think retry is orthogonal to this policy. serve answers "what about the rest of the proxy?" while retry answers "what about the failed cluster?" They compose independently. By making onVirtualClusterFailure a structured block rather than a flat string, new keys can be added alongside serve in future proposals without breaking existing configs. For example:

proxy:
  onVirtualClusterFailure:
    serve: remaining
    # future extensions — new keys, existing configs unaffected:
    # retry:
    #   maxRetries: 3
    #   backoff: 5s
    # escalate:
    #   url: http://example.slack.com/
    #   messageFormat: "%s failed with %s"
    #   args: [VIRTUAL_CLUSTER, EXCEPTION_MESSAGE]

This proposal defines serve with its two values. The rejected alternative on automatic retry still applies — we're not committing to implementing retry here, but the config structure leaves room for it without a breaking change.

[tombentley]

The proxy cannot know when/whether the client receives a response that's been forwarded to it (at least not without inferring that fact from the receipt of another request, but that defeats the point).

In any case, I just don't see that this can really be completely solved by the proxy in its current shape. To solve it properly I think we'd need to "virtualise the brokers". That would allow the proxy to leverage Kafka's own technique for shutting down brokers cleanly: Tell the client that this broker is no longer the leader or coordinator for anything, so that all the clients update their metadata to find the new leader/coordinator. Doing that would require a proper cluster-of-proxies, which is not something we have today.

So I think what you're describing here is not perfect, but it's as good are we're likely to get for now, and we shouldn't block this proposal while we figure out a more perfect solution.

[SamBarker]

Agreed on both points. "As best it can" now hedges the in-flight definition — the proxy observes its own reads and writes, not what the client has actually sent or received.

And yes, virtualising the brokers would let the proxy leverage Kafka's own clean shutdown mechanism (leader migration via metadata updates). That would be a really compelling capability if we can figure out how to make it work. For now, best-effort draining with a timeout backstop is as good as we can get without it.

[Uzziee]

Can you help me understand how will the experience look with rollback on failure for hot-reload ?

This is what I think would happen if we mark it as "remaining"

Eg:-

1. There are 3 virtual clusters running (cluster-1, cluster-2, cluster-3)
2. We apply a change to cluster-2
3. cluster-2 fails to come up, initiate rollback
4. cluster-1, cluster-3 will still continue to serve the traffic during this rollback phase
5. cluster-2 is rolled back to previous state and is up and running

The only risk I see with "remaining" is during the app startup, we will have to put in some extra logic to ensure that the entire start state is stable