Appearance

HA, etcd quorum, and failure domains

Keep API availability distinct from storage consensus and workload survival.

What you will understand

  • Calculate quorum from configured voting membership.
  • Explain failure-domain placement and timing constraints.
  • Distinguish a surviving workload from a healthy control plane.

Builds on: Leases and leader election, Capacity and scale trade-offs

Mental model: multiple API servers still share one coordination system

A highly available control plane commonly has multiple API servers behind an appropriate endpoint, redundant scheduler and controller-manager processes using leader election, and an etcd deployment capable of maintaining quorum through the intended failures. API server replicas help only while their dependencies and routing remain available. Three API servers pointing at one unavailable etcd member do not provide three independent durable copies of cluster state.

etcd uses consensus among its configured voting members. A three-voter cluster needs two votes; a five-voter cluster needs three. Losing one voter from a three-member cluster can preserve quorum, while losing two cannot. Count configured voting membership, not merely the processes currently responding. Adding a fourth voting member changes the majority requirement to three and does not increase the number of tolerated voter failures compared with three. More members also add replication and operational cost.

Quorum counts configured voters

Writes can commit only with the required healthy agreement; extra API servers do not replace votes.

Cluster viewStep 1 of 3
Three configured votersThe voting membership contains A, B and C, so quorum is two.. API servers, Read/write API boundary, idle. etcd voter A, Configured member, active. etcd voter B, Configured member, idle. etcd voter C, Configured member, idle. Committed state, Majority required, idleAPI servers: Read/write API boundary
API serversRead/write API boundaryContext
etcd voter A: Configured member
etcd voter AConfigured memberIn focus
etcd voter B: Configured member
etcd voter BConfigured memberContext
etcd voter C: Configured member
etcd voter CConfigured memberContext
Committed state: Majority required
Committed stateMajority requiredContext
API serversRead/write API boundaryContext
etcd voter AConfigured memberIn focus
etcd voter BConfigured memberContext
etcd voter CConfigured memberContext
Committed stateMajority requiredContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Two voters lost
Three configured voters

The voting membership contains A, B and C, so quorum is two.

Read the complete visual sequence
  1. Three configured voters

    The voting membership contains A, B and C, so quorum is two.

  2. One voter unavailable

    A and B can still form the configured majority while C is unavailable.

  3. Commit with majority

    Writes can commit only with the required healthy agreement; extra API servers do not replace votes.

Two voters lost

A lone survivor cannot declare itself the majority of a three-voter cluster.

  1. B and C unavailable

    The configured voting membership still contains all three until valid membership changes commit.

  2. Only A remains

    One vote is less than the required two, so normal consensus writes cannot proceed.

  3. Plan recovery deliberately

    Recover members or follow verified disaster recovery; do not count only surviving processes.

About this model

Voters are simplified to three members; learners, read consistency modes and detailed Raft messages are omitted.

Walkthrough: topology and timing can defeat apparent redundancy

Stacked kubeadm topology places etcd members with control-plane nodes, while external etcd separates their hosts and lifecycle. Both require intentional failure-domain planning. Three virtual machines on one physical host or power domain can fail together. Spreading across distant regions may improve geographic separation while introducing latency that harms consensus and operational responsiveness. The right topology follows tested latency, durability and availability requirements rather than a rule that the most distant members are always safest.

Consensus commits depend on network round trips and durable disk behavior. Slow storage, overloaded hosts or packet loss can produce request latency and leadership churn without a clean node-down event. Election timeouts must accommodate the environment; shortening them cannot make an unreliable network reliable. Membership changes are also writes that require a healthy enough cluster to commit. Plan replacements sequentially using the supported member/learner workflow rather than removing several voters and hoping new machines restore quorum automatically.

Redundancy requires independent failure domains

Existing Pods may serve during an API outage, while scheduling and reconciliation are impaired.

Cluster viewStep 1 of 3
Count domains, not iconsTwo replicas in one failure domain can disappear in the same infrastructure incident.. Shared host / zone, Possible correlated failure, active. Control plane A, API + stacked etcd, idle. Control plane B, API + stacked etcd, idle. Disk and network, Consensus latency path, idle. Existing workloads, May continue without API, idleShared host / zone: Possible correlated failure
Shared host / zonePossible correlated failureIn focus
Control plane A: API + stacked etcd
Control plane AAPI + stacked etcdContext
Control plane B: API + stacked etcd
Control plane BAPI + stacked etcdContext
Disk and network: Consensus latency path
Disk and networkConsensus latency pathContext
Existing workloads: May continue without API
Existing workloadsMay continue without APIContext
Shared host / zonePossible correlated failureIn focus
Control plane AAPI + stacked etcdContext
Control plane BAPI + stacked etcdContext
Disk and networkConsensus latency pathContext
Existing workloadsMay continue without APIContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Correlated outage
Count domains, not icons

Two replicas in one failure domain can disappear in the same infrastructure incident.

Read the complete visual sequence
  1. Count domains, not icons

    Two replicas in one failure domain can disappear in the same infrastructure incident.

  2. Measure the commit path

    Network and disk latency affect consensus and election behavior even when processes remain Running.

  3. Separate service health

    Existing Pods may serve during an API outage, while scheduling and reconciliation are impaired.

Correlated outage

All apparent replicas rely on one shared failure source.

  1. Shared infrastructure fails

    A host, zone or storage dependency removes multiple colocated members together.

  2. Replica count was misleading

    Multiple control-plane processes did not provide independent failure tolerance.

  3. Workload continuity is conditional

    Some existing traffic can continue, but this does not prove a healthy control plane or guaranteed service survival.

About this model

The figure illustrates correlated dependencies rather than prescribing one universal cloud topology.

This ConfigMap is a small API marker for the backup exercise. It demonstrates that durable API state belongs in the control-plane recovery plan; it does not configure an HA etcd cluster.

apiVersion: v1
kind: ConfigMap
metadata:
  name: m18-marker
  namespace: kv-lab
  labels:
    lab: m18
data:
  purpose: control-plane-recovery-check
kubectl --context kind-kv-lab get --raw='/readyz?verbose'
kubectl --context kind-kv-lab -n kube-system get pods -o wide
kubectl --context kind-kv-lab -n kube-system get leases
# Read-only evidence: the single-node teaching cluster is not HA.
kubectl --context kind-kv-lab get nodes

Practice: reason about a failure without removing a voter

Use the m18 local exercise to capture a real API-state snapshot in the named disposable cluster. The standard local topology has a single control-plane node; it intentionally does not demonstrate a quorum surviving machine loss. Draw three voters A, B and C across three failure domains. Predict the outcome when C fails, then when B is isolated from A. Distinguish the configured majority from the number of containers whose process appears Running.

Now draw worker Pods that are already serving traffic. If the API becomes unavailable, some existing workloads can continue while reconciliation, scheduling and new writes are disrupted. That continuing traffic is useful but not proof that the cluster remains operationally healthy. Node failures, expiring credentials, network changes or application dependencies can still affect those workloads. Define the control-plane and application health checks separately in an incident report.

Pitfalls: restoring quorum is not ordinary scaling

Do not run membership removal or force-new-cluster commands as a generic response to a failed endpoint. Verify which members are alive, which data directories belong to the cluster, and whether quorum can be recovered through an ordinary member restart. Permanent majority loss requires a carefully planned recovery path, commonly from a verified snapshot, with the old cluster isolated to avoid competing histories. Back up before planned changes and test the exact etcd version’s documented procedure.

A load balancer can mask one failed API server, but a readiness check should remove unhealthy backends rather than repeatedly route writes to them. Certificate identities and advertised addresses must match the topology. A control-plane plan should include endpoint failover, member replacement, disk capacity, monitoring and human access when the primary API is impaired.

Check your understanding

Only two processes respond in a configured five-voter etcd cluster. Do they form a majority of the survivors?

They are a majority of the responders but not a quorum of the configured voting membership. Five voters require three votes. Counting only healthy processes would violate the consensus membership boundary.

Next, maintain the identities and version contracts that let these components communicate. Redundant replicas cannot compensate for expired certificates or an unsupported upgrade sequence.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.