HA, etcd quorum, and failure domains
Keep API availability distinct from storage consensus and workload survival.
What you will understand
- Calculate quorum from configured voting membership.
- Explain failure-domain placement and timing constraints.
- Distinguish a surviving workload from a healthy control plane.
Builds on: Leases and leader election, Capacity and scale trade-offs
Mental model: multiple API servers still share one coordination system
A highly available control plane commonly has multiple API servers behind an appropriate endpoint, redundant scheduler and controller-manager processes using leader election, and an etcd deployment capable of maintaining quorum through the intended failures. API server replicas help only while their dependencies and routing remain available. Three API servers pointing at one unavailable etcd member do not provide three independent durable copies of cluster state.
etcd uses consensus among its configured voting members. A three-voter cluster needs two votes; a five-voter cluster needs three. Losing one voter from a three-member cluster can preserve quorum, while losing two cannot. Count configured voting membership, not merely the processes currently responding. Adding a fourth voting member changes the majority requirement to three and does not increase the number of tolerated voter failures compared with three. More members also add replication and operational cost.
Quorum counts configured voters
Writes can commit only with the required healthy agreement; extra API servers do not replace votes.
The voting membership contains A, B and C, so quorum is two.
- API servers to etcd voter Arequest
- etcd voter A to etcd voter Breplication
- etcd voter A to etcd voter Creplication
Read the complete visual sequence
- Three configured voters
The voting membership contains A, B and C, so quorum is two.
- One voter unavailable
A and B can still form the configured majority while C is unavailable.
- Commit with majority
Writes can commit only with the required healthy agreement; extra API servers do not replace votes.
Two voters lost
A lone survivor cannot declare itself the majority of a three-voter cluster.
- B and C unavailable
The configured voting membership still contains all three until valid membership changes commit.
- Only A remains
One vote is less than the required two, so normal consensus writes cannot proceed.
- Plan recovery deliberately
Recover members or follow verified disaster recovery; do not count only surviving processes.
About this model
Voters are simplified to three members; learners, read consistency modes and detailed Raft messages are omitted.
Walkthrough: topology and timing can defeat apparent redundancy
Stacked kubeadm topology places etcd members with control-plane nodes, while external etcd separates their hosts and lifecycle. Both require intentional failure-domain planning. Three virtual machines on one physical host or power domain can fail together. Spreading across distant regions may improve geographic separation while introducing latency that harms consensus and operational responsiveness. The right topology follows tested latency, durability and availability requirements rather than a rule that the most distant members are always safest.
Consensus commits depend on network round trips and durable disk behavior. Slow storage, overloaded hosts or packet loss can produce request latency and leadership churn without a clean node-down event. Election timeouts must accommodate the environment; shortening them cannot make an unreliable network reliable. Membership changes are also writes that require a healthy enough cluster to commit. Plan replacements sequentially using the supported member/learner workflow rather than removing several voters and hoping new machines restore quorum automatically.
Redundancy requires independent failure domains
Existing Pods may serve during an API outage, while scheduling and reconciliation are impaired.
Two replicas in one failure domain can disappear in the same infrastructure incident.
- Shared host / zone to Control plane Acorrelated placement
- Shared host / zone to Control plane Bcorrelated placement
Read the complete visual sequence
- Count domains, not icons
Two replicas in one failure domain can disappear in the same infrastructure incident.
- Measure the commit path
Network and disk latency affect consensus and election behavior even when processes remain Running.
- Separate service health
Existing Pods may serve during an API outage, while scheduling and reconciliation are impaired.
Correlated outage
All apparent replicas rely on one shared failure source.
- Shared infrastructure fails
A host, zone or storage dependency removes multiple colocated members together.
- Replica count was misleading
Multiple control-plane processes did not provide independent failure tolerance.
- Workload continuity is conditional
Some existing traffic can continue, but this does not prove a healthy control plane or guaranteed service survival.
About this model
The figure illustrates correlated dependencies rather than prescribing one universal cloud topology.
This ConfigMap is a small API marker for the backup exercise. It demonstrates that durable API state belongs in the control-plane recovery plan; it does not configure an HA etcd cluster.
apiVersion: v1
kind: ConfigMap
metadata:
name: m18-marker
namespace: kv-lab
labels:
lab: m18
data:
purpose: control-plane-recovery-check
kubectl --context kind-kv-lab get --raw='/readyz?verbose'
kubectl --context kind-kv-lab -n kube-system get pods -o wide
kubectl --context kind-kv-lab -n kube-system get leases
# Read-only evidence: the single-node teaching cluster is not HA.
kubectl --context kind-kv-lab get nodes
Practice: reason about a failure without removing a voter
Use the m18 local exercise to capture a real API-state snapshot in the named disposable cluster. The standard local topology has a single control-plane node; it intentionally does not demonstrate a quorum surviving machine loss. Draw three voters A, B and C across three failure domains. Predict the outcome when C fails, then when B is isolated from A. Distinguish the configured majority from the number of containers whose process appears Running.
Now draw worker Pods that are already serving traffic. If the API becomes unavailable, some existing workloads can continue while reconciliation, scheduling and new writes are disrupted. That continuing traffic is useful but not proof that the cluster remains operationally healthy. Node failures, expiring credentials, network changes or application dependencies can still affect those workloads. Define the control-plane and application health checks separately in an incident report.
Pitfalls: restoring quorum is not ordinary scaling
Do not run membership removal or force-new-cluster commands as a generic response to a failed endpoint. Verify which members are alive, which data directories belong to the cluster, and whether quorum can be recovered through an ordinary member restart. Permanent majority loss requires a carefully planned recovery path, commonly from a verified snapshot, with the old cluster isolated to avoid competing histories. Back up before planned changes and test the exact etcd version’s documented procedure.
A load balancer can mask one failed API server, but a readiness check should remove unhealthy backends rather than repeatedly route writes to them. Certificate identities and advertised addresses must match the topology. A control-plane plan should include endpoint failover, member replacement, disk capacity, monitoring and human access when the primary API is impaired.
Check your understanding
Only two processes respond in a configured five-voter etcd cluster. Do they form a majority of the survivors?
They are a majority of the responders but not a quorum of the configured voting membership. Five voters require three votes. Counting only healthy processes would violate the consensus membership boundary.
Next, maintain the identities and version contracts that let these components communicate. Redundant replicas cannot compensate for expired certificates or an unsupported upgrade sequence.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise