Appearance

StatefulSets and stable identity

Use ordered identities and per-replica claims while keeping application replication explicit.

What you will understand

  • Explain StatefulSet ordinals, stable names and headless discovery
  • Trace per-replica volumeClaimTemplates
  • Predict ordered rollout blocking and safe recovery choices

Builds on: Snapshots and storage recovery

Mental model: stable logical identities over disposable processes

A StatefulSet manages Pods with stable ordinal identities such as ledger-0, ledger-1 and ledger-2. Replacing ledger-1 creates a new Pod object with the same logical name and its associated storage relationship. Its UID and process instance are new. Stable identity helps software that distinguishes peers or associates durable data with a specific replica. It does not keep the original process alive and does not itself replicate database records.

A headless Service, using clusterIP: None, provides discovery without a single load-balancing ClusterIP. Combined with the StatefulSet’s serviceName, it supports predictable per-Pod DNS names such as ledger-0.ledger. DNS answers depend on readiness and publication configuration, so bootstrap-sensitive applications must account for when peers become discoverable. Negative DNS caching can also delay discovery of a newly created name. Stable naming is a coordination tool, not a promise that a hostname always resolves to a healthy process.

An ordinal survives replacement; a process does not

Trace StatefulSet naming through a headless Service and one Pod replacement.

Ownership viewStep 1 of 4
Assign logical membersThe controller creates predictable ordinal names instead of interchangeable random Deployment identities.. StatefulSet ledger, replicas=3; serviceName=ledger, active. Headless Service, clusterIP: None, idle. ledger-0, Ordinal 0, ready. ledger-1, Ordinal 1, ready. ledger-2, Ordinal 2, ready. DNS, Per-Pod stable names, idleStatefulSet ledger: replicas=3; serviceName=ledger
StatefulSet ledgerreplicas=3; serviceName=ledgerIn focus
Headless Service: clusterIP: None
Headless ServiceclusterIP: NoneContext
ledger-0: Ordinal 0
ledger-0Ordinal 0Ready
ledger-1: Ordinal 1
ledger-1Ordinal 1Ready
ledger-2: Ordinal 2
ledger-2Ordinal 2Ready
DNS: Per-Pod stable names
DNSPer-Pod stable namesContext
StatefulSet ledgerreplicas=3; serviceName=ledgerIn focus
Headless ServiceclusterIP: NoneContext
ledger-0Ordinal 0Ready
ledger-1Ordinal 1Ready
ledger-2Ordinal 2Ready
DNSPer-Pod stable namesContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
DNS waits for readiness
Assign logical members

The controller creates predictable ordinal names instead of interchangeable random Deployment identities.

Read the complete visual sequence
  1. Assign logical members

    The controller creates predictable ordinal names instead of interchangeable random Deployment identities.

  2. Publish peer discovery

    The headless Service and StatefulSet subdomain support member names. DNS publication still depends on endpoint readiness settings.

  3. Lose one process

    The old ledger-1 Pod ends. Its logical ordinal remains part of the desired state even though that Pod UID is gone.

  4. Create a new member instance

    The replacement is again ledger-1, with a new UID and potentially a new IP. Clients resolve the stable name to current addresses.

DNS waits for readiness

A bootstrap process expects a peer name before the Service publishes it.

  1. Member exists but is unready

    Running does not necessarily make the member ready for ordinary Service discovery.

  2. Name not yet usable

    The client cannot rely on ordinary ready endpoint publication to discover this unready member.

  3. Use application-aware bootstrap

    Evaluate readiness dependencies and publishNotReadyAddresses deliberately when the application requires bootstrap discovery; account for DNS caching.

About this model

Default ordinals starting at zero and a matching same-namespace headless Service.

Walkthrough: identity and data travel together

This optional example shows the essential fields of a claim-backed StatefulSet. To run it, supply a matching headless Service named ledger and a functioning default StorageClass. The module lab’s baseline instead runs a one-member m10 StatefulSet with a headless Service and no claims; it demonstrates identity, while the m09 lab separately demonstrates PVC persistence. The file writer below combines those concepts without implementing a replicated database.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ledger
spec:
  serviceName: ledger        # Matching headless Service must exist.
  replicas: 3
  selector:
    matchLabels: {app: ledger}
  template:
    metadata:
      labels: {app: ledger}
    spec:
      containers:
        - name: writer
          image: busybox:1.36
          command: ["sh", "-c", "hostname >> /data/starts; sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1Gi

Each ordinal gets its own claim, such as data-ledger-0. The claim belongs to that logical replica; the StatefulSet does not share one generic disk across all members. A replacement mounts the same claim, subject to driver attachment and topology constraints. Default retention generally preserves claims when Pods are removed or the StatefulSet is deleted. Explicit PVC retention policy can alter scale-down and deletion behavior, so read the actual policy before treating cleanup as harmless.

OrderedReady, the default Pod management policy, creates and scales members in ordinal order and waits for readiness before proceeding. Default rolling updates replace higher ordinals before lower ones, waiting for readiness. This ordering supports some stateful applications’ membership protocols, but can also block progress when a member never becomes ready. Parallel changes scale behavior, not a universal removal of every rollout ordering guarantee. Read the controller’s chosen update strategy rather than inferring it from Pod count alone.

Each member keeps a distinct claim and rollout checkpoint

Follow volumeClaimTemplates into per-ordinal claims and a readiness-gated rolling update.

Storage viewStep 1 of 4
Instantiate claimsThe template produces distinct claims such as data-ledger-0. The controller is not cloning one shared writable filesystem for every Pod.. StatefulSet, volumeClaimTemplates: data, active. ledger-0, Lower ordinal, idle. ledger-1, Middle ordinal, idle. ledger-2, Highest ordinal, idle. Claims 0 / 1 / 2, One claim per ordinal, ready. Readiness checkpoint, Wait before next replacement, idleStatefulSet: volumeClaimTemplates: data
StatefulSetvolumeClaimTemplates: dataIn focus
ledger-0: Lower ordinal
ledger-0Lower ordinalContext
ledger-1: Middle ordinal
ledger-1Middle ordinalContext
ledger-2: Highest ordinal
ledger-2Highest ordinalContext
Claims 0 / 1 / 2: One claim per ordinal
Claims 0 / 1 / 2One claim per ordinalReady
Readiness checkpoint: Wait before next replacement
Readiness checkpointWait before next replacementContext
StatefulSetvolumeClaimTemplates: dataIn focus
ledger-0Lower ordinalContext
ledger-1Middle ordinalContext
ledger-2Highest ordinalContext
Claims 0 / 1 / 2One claim per ordinalReady
Readiness checkpointWait before next replacementContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Updated member never becomes ready
Instantiate claims

The template produces distinct claims such as data-ledger-0. The controller is not cloning one shared writable filesystem for every Pod.

Read the complete visual sequence
  1. Instantiate claims

    The template produces distinct claims such as data-ledger-0. The controller is not cloning one shared writable filesystem for every Pod.

  2. Reattach the same member data

    When ledger-1 is replaced, its new instance references the ordinal-1 claim rather than another member disk.

  3. Update highest ordinal first

    A default rolling update replaces ledger-2 before lower ordinals and observes its readiness checkpoint.

  4. Proceed after readiness

    When the updated highest member is ready, the controller can update the next lower member. Application replication remains the application responsibility.

Updated member never becomes ready

Default ordered update stops before replacing more members.

  1. New member fails its probe

    ledger-2 starts but cannot satisfy readiness due to application or storage configuration.

  2. Lower members remain old

    The readiness checkpoint prevents further replacement, preserving lower members while the fault is investigated.

  3. Inspect before force recovery

    Read logs, events, template revision and application state. Follow documented rollback and member recovery rather than deleting all claims.

About this model

Default OrderedReady management and default rolling-update behavior; retained PVCs.

Practice: replace one member and inspect continuity

kubectl get statefulset ledger
kubectl get pods -l app=ledger -o wide
kubectl get pvc
kubectl exec ledger-1 -- cat /data/starts
kubectl delete pod ledger-1
kubectl get pod ledger-1 -w
kubectl exec ledger-1 -- cat /data/starts

Run these commands only after creating the optional three-member ledger example in a disposable namespace; the baseline workload lab does not create ledger-1 or /data/starts. Expect a replacement named ledger-1 and another line in the file on its retained claim. Compare its new UID to the previous one. Inspect the headless Service and per-Pod DNS from a client. The data demonstration proves claim reuse for this local setup; it does not prove failover across a lost host or recovery from corrupted application files.

For a blocked rollout, inspect the member’s readiness probe, events, application logs and mounted volume before changing ordering. A rollback can restore the template while a bad member still needs targeted recovery. Do not force-delete a member on an unreachable node until you have established that its old process cannot still write to shared state. Duplicate logical identities with active writers are much more dangerous than a visibly stalled replacement.

Pitfalls: controller ordering is not a database protocol

A three-replica StatefulSet does not elect a leader, create a quorum or copy data unless the application implements those behaviors. A readiness probe should reflect that application’s serving role, and startup dependencies must avoid circular waits where every member refuses readiness until all others are ready. Choose an operator or application-specific procedure when membership, upgrade and recovery require more than generic Pod identity management.

Check your understanding

After ledger-1 is replaced, must it have the same UID, IP address and data?

Its logical Pod name is stable, but its UID is new and its IP can change. Its data can continue through the retained per-ordinal claim if the storage remains available and the replacement mounts it successfully. Identity continuity and data availability are distinct guarantees, and neither automatically establishes healthy application replication.

Next, compare this identity-oriented controller with node-local DaemonSets and finite Jobs. The controller choice should match what a replica means for the application.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.