[BUG] Failure zones of control planes shouldn't depend on location of worker nodes

[janiskemper]

In https://docs.scs.community/standards/scs-0214-v2-k8s-node-distribution we state the following:

At least one control plane instance MUST be run in each "failure zone" used for the cluster, more instances per "failure zone" are possible to provide fault-tolerance inside a zone.

This doesn't make sense IMO, because this essentially creates a condition on the control plane, based on the worker nodes. If I create a new worker node in a different failure zone, suddenly my control planes have to change and I need another control plane in this new failure zone.

If I use hypothetically 4 different failure zones for workers, I would have to have 5 control planes (4 is not possible). If I have only 3 different failure zones for my workers, 3 control planes would be enough.

This behavior is not practical and realistic in my opinion and should be left to the Kubernetes user designing the desired architecture.

Having multiple Kubernetes clusters is for example a better architecture often than one Kubernetes cluster spread over many failure zones / regions.

I therefore would argue that this sentence should be removed from the standard.

This has been already mentioned as outcome of the KaaS Hackathon.

[janiskemper]

@mbuechse @depressiveRobot

[mbuechse]

this essentially creates a condition on the control plane, based on the worker nodes

Not sure if I agree. I think it depends on how you interpret the word "used" in

At least one control plane instance MUST be run in each "failure zone" used for the cluster

In my interpretation, this does not mean "actually" or "currently" used, rather more like "usable in principle". So it would not be a dynamic condition.

The idea here is that you advertise that your cluster has access to three failure zones, and then a control-plane node should be present in each of them, so that the resilience of the cluster matches the expectation from this advertisement.

[janiskemper]

"Each failure zone used by the cluster" is a very straightforward formulation IMO.

Imagine I wanna have worker nodes in two locations of Hetzner, Falkenstein and Helsinki, then I would also need to put the control planes in Falkenstein and Helsinki. If I change my opinion and have my workers only in Falkenstein, then this would be the only one used, so I can remove my control plane in Helsinki and put it to Falkenstein.

It wouldn't be any better if we say either "control planes must be in different failure zones", because that would mean that for Hetzner you would have to spread them across minimum Germany and Finland in case you have 3 control planes.

I think that this formulation is either terribly misleading (as you suggest @mbuechse), or wrong (as I believe).

In either case, I don't think it adds any value. The official Kubernetes resources linked in the document already state that for a "large cluster" you should ideally first scale control planes across failure zones as best practice (see https://kubernetes.io/docs/setup/best-practices/cluster-large/#control-plane-components).

If anything, we tried to rewrite this formulation from the Kubernetes docs, which is not necessary, since we link them already.

A very concrete last statement on why I am very firm that we need to AT LEAST get rid of the "must" here:

Usually, we tell customers that if they wanna use multiple locations of Hetzner (they don't have other failure zones), that they should have one cluster per location. If they do, then there is only one failure zone and we are compliant with this standard.

However, if the customer decided to add workers in another location, we wouldn't be compliant anymore, because the workers in the cluster use two different locations, but we cannot move one control plane to another location.. That's impossible with the load balancing solution.

Therefore: We would be only compliant at all times with this standard if we actively forbid our customers to use multiple locations for workers. That doesn't make sense IMO and cannot be the intention behind this.

[mbuechse]

Oh. I did take a wrong turn here. I'm just puzzled that the customer would be able to configure their cluster in that way even after it has been created. At any rate, if the customer actively constructs a cluster that violates the standard, then I suppose that's on them. I think in case of Hetzner, if you cannot have a control plane spanning multiple locations, then the customers should indeed be told that spanning workers over multiple locations is an edge case that they use at their own risk and that is definitely not supported within the realm of SCS.

[mbuechse]

What I mean to say here: if the customer is told what I just said, you're off the hook.

[janiskemper]

I think one aspect that I haven't mentioned explicitly but that is "killing it" is the "failure zone" vs "region" topic.

I think it is not at all meant here that you have a cluster with control planes spanning multiple countries. I think that no provider would really do that (not sure how hyperscalers do it internally). Therefore, it cannot be a goal to have control planes in multiple regions.

At the same time, if you have the cluster in one datacenter, with one datacenter having multiple failure zones, then there is an advantage obviously.

I still think that this phrase cannot be formulated like this (and is not formulated like this in the original document I linked also again), because of this dynamic behavior, where the customer can start using new failure zones and which would mean the control planes have to adapt afterwards.

I would therefore still say the best idea is to remove that one sentence, because also it is already mentioned in the Kubernetes docs as something you should ideally do - scale control planes in different failure zones.

WDYT?

[neuroserve]

AFAIK, the only "hard" dependency is the latency between the etcd instances (10 ms max, IIRC). All other components of the cluster have much more relaxed requirements regarding network RTTs. If you can establish an etcd cluster in three datacenters (of one region, if we consider the definitions of the hyperscalers), you should be able to put your other cluster components - especially worker nodes - in more remote locations. See https://clastix.io/post/bridging-the-gap-hybrid-kubernetes-clusters-with-remote-control-planes/.

[depressiveRobot]

@benedikt-haug @fzakfeld What do you think?

[fzakfeld]

I unfortunately don't have experience with spanning clusters across regions, cities or data centres. We (ScaleUp) currently don't offer multiple AZs within one region, but once we do it would make sense to span our seed clusters, where the control planes are hosted across those AZs as well.

We've had customers in the past that ran clusters in different regions than their control plane (for what ever reason) which caused some issues with latency and reliability, so I would advice against it unless someone can prove that this works in a reliable way

[mbuechse]

FYI I am working on a PR