StatefulSets and stable identity
Use ordered identities and per-replica claims while keeping application replication explicit.
What you will understand
- Explain StatefulSet ordinals, stable names and headless discovery
- Trace per-replica volumeClaimTemplates
- Predict ordered rollout blocking and safe recovery choices
Builds on: Snapshots and storage recovery
Mental model: stable logical identities over disposable processes
A StatefulSet manages Pods with stable ordinal identities such as ledger-0, ledger-1 and ledger-2. Replacing ledger-1 creates a new Pod object with the same logical name and its associated storage relationship. Its UID and process instance are new. Stable identity helps software that distinguishes peers or associates durable data with a specific replica. It does not keep the original process alive and does not itself replicate database records.
A headless Service, using clusterIP: None, provides discovery without a single load-balancing ClusterIP. Combined with the StatefulSet’s serviceName, it supports predictable per-Pod DNS names such as ledger-0.ledger. DNS answers depend on readiness and publication configuration, so bootstrap-sensitive applications must account for when peers become discoverable. Negative DNS caching can also delay discovery of a newly created name. Stable naming is a coordination tool, not a promise that a hostname always resolves to a healthy process.
An ordinal survives replacement; a process does not
Trace StatefulSet naming through a headless Service and one Pod replacement.
The controller creates predictable ordinal names instead of interchangeable random Deployment identities.
- StatefulSet ledger to ledger-0ordinal 0
- StatefulSet ledger to ledger-1ordinal 1
- StatefulSet ledger to ledger-2ordinal 2
Read the complete visual sequence
- Assign logical members
The controller creates predictable ordinal names instead of interchangeable random Deployment identities.
- Publish peer discovery
The headless Service and StatefulSet subdomain support member names. DNS publication still depends on endpoint readiness settings.
- Lose one process
The old ledger-1 Pod ends. Its logical ordinal remains part of the desired state even though that Pod UID is gone.
- Create a new member instance
The replacement is again ledger-1, with a new UID and potentially a new IP. Clients resolve the stable name to current addresses.
DNS waits for readiness
A bootstrap process expects a peer name before the Service publishes it.
- Member exists but is unready
Running does not necessarily make the member ready for ordinary Service discovery.
- Name not yet usable
The client cannot rely on ordinary ready endpoint publication to discover this unready member.
- Use application-aware bootstrap
Evaluate readiness dependencies and publishNotReadyAddresses deliberately when the application requires bootstrap discovery; account for DNS caching.
About this model
Default ordinals starting at zero and a matching same-namespace headless Service.
Walkthrough: identity and data travel together
This optional example shows the essential fields of a claim-backed StatefulSet. To run it, supply a matching headless Service named ledger and a functioning default StorageClass. The module lab’s baseline instead runs a one-member m10 StatefulSet with a headless Service and no claims; it demonstrates identity, while the m09 lab separately demonstrates PVC persistence. The file writer below combines those concepts without implementing a replicated database.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ledger
spec:
serviceName: ledger # Matching headless Service must exist.
replicas: 3
selector:
matchLabels: {app: ledger}
template:
metadata:
labels: {app: ledger}
spec:
containers:
- name: writer
image: busybox:1.36
command: ["sh", "-c", "hostname >> /data/starts; sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 1Gi
Each ordinal gets its own claim, such as data-ledger-0. The claim belongs to that logical replica; the StatefulSet does not share one generic disk across all members. A replacement mounts the same claim, subject to driver attachment and topology constraints. Default retention generally preserves claims when Pods are removed or the StatefulSet is deleted. Explicit PVC retention policy can alter scale-down and deletion behavior, so read the actual policy before treating cleanup as harmless.
OrderedReady, the default Pod management policy, creates and scales members in ordinal order and waits for readiness before proceeding. Default rolling updates replace higher ordinals before lower ones, waiting for readiness. This ordering supports some stateful applications’ membership protocols, but can also block progress when a member never becomes ready. Parallel changes scale behavior, not a universal removal of every rollout ordering guarantee. Read the controller’s chosen update strategy rather than inferring it from Pod count alone.
Each member keeps a distinct claim and rollout checkpoint
Follow volumeClaimTemplates into per-ordinal claims and a readiness-gated rolling update.
The template produces distinct claims such as data-ledger-0. The controller is not cloning one shared writable filesystem for every Pod.
- StatefulSet to Claims 0 / 1 / 2per-ordinal PVCs
Read the complete visual sequence
- Instantiate claims
The template produces distinct claims such as data-ledger-0. The controller is not cloning one shared writable filesystem for every Pod.
- Reattach the same member data
When ledger-1 is replaced, its new instance references the ordinal-1 claim rather than another member disk.
- Update highest ordinal first
A default rolling update replaces ledger-2 before lower ordinals and observes its readiness checkpoint.
- Proceed after readiness
When the updated highest member is ready, the controller can update the next lower member. Application replication remains the application responsibility.
Updated member never becomes ready
Default ordered update stops before replacing more members.
- New member fails its probe
ledger-2 starts but cannot satisfy readiness due to application or storage configuration.
- Lower members remain old
The readiness checkpoint prevents further replacement, preserving lower members while the fault is investigated.
- Inspect before force recovery
Read logs, events, template revision and application state. Follow documented rollback and member recovery rather than deleting all claims.
About this model
Default OrderedReady management and default rolling-update behavior; retained PVCs.
Practice: replace one member and inspect continuity
kubectl get statefulset ledger
kubectl get pods -l app=ledger -o wide
kubectl get pvc
kubectl exec ledger-1 -- cat /data/starts
kubectl delete pod ledger-1
kubectl get pod ledger-1 -w
kubectl exec ledger-1 -- cat /data/starts
Run these commands only after creating the optional three-member ledger example in a disposable namespace; the baseline workload lab does not create ledger-1 or /data/starts. Expect a replacement named ledger-1 and another line in the file on its retained claim. Compare its new UID to the previous one. Inspect the headless Service and per-Pod DNS from a client. The data demonstration proves claim reuse for this local setup; it does not prove failover across a lost host or recovery from corrupted application files.
For a blocked rollout, inspect the member’s readiness probe, events, application logs and mounted volume before changing ordering. A rollback can restore the template while a bad member still needs targeted recovery. Do not force-delete a member on an unreachable node until you have established that its old process cannot still write to shared state. Duplicate logical identities with active writers are much more dangerous than a visibly stalled replacement.
Pitfalls: controller ordering is not a database protocol
A three-replica StatefulSet does not elect a leader, create a quorum or copy data unless the application implements those behaviors. A readiness probe should reflect that application’s serving role, and startup dependencies must avoid circular waits where every member refuses readiness until all others are ready. Choose an operator or application-specific procedure when membership, upgrade and recovery require more than generic Pod identity management.
Check your understanding
After ledger-1 is replaced, must it have the same UID, IP address and data?
Its logical Pod name is stable, but its UID is new and its IP can change. Its data can continue through the retained per-ordinal claim if the storage remains available and the replacement mounts it successfully. Identity continuity and data availability are distinct guarantees, and neither automatically establishes healthy application replication.
Next, compare this identity-oriented controller with node-local DaemonSets and finite Jobs. The controller choice should match what a replica means for the application.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise