Appearance

Capstone: recover stateful data

Back up a quiescent file, delete it, restore it, and prove persistence.

What you will understand

  • Separate Pod replacement from restoration of deleted records.
  • Verify an external backup and restored contents.
  • State the consistency boundary and limits of the file-based exercise.

Builds on: Termination, disruption, and drain, Backup, restore, and disaster recovery, Capstone: recover a stateless app

Mental model: persistence also preserves mistakes

A PersistentVolumeClaim lets a replacement Pod reconnect to storage beyond the old Pod’s lifetime. This preserves useful files, but it also preserves an accidental deletion. Deleting and recreating the consumer Pod does not recreate a file removed from its mounted filesystem. Recovery needs a retained copy from before the loss and a procedure that restores it. This distinction is the central failure boundary of the stateful capstone.

The exercise writes one small CSV record to a PVC, captures a backup in the private host directory selected by backup-path.py, deliberately deletes the file, restores it with kubectl cp, and checks exact contents. It then replaces the Pod and checks the restored record again. The backup exists outside the cluster so destruction of the disposable cluster would not automatically remove that host file. Protect and clean up the host copy intentionally as part of the exercise lifecycle.

Persistence preserves deletion as well as data

The script removes the file from the mounted PVC and verifies absence; a Pod restart cannot undo this deletion.

Storage viewStep 1 of 3
Write then quiesceThe single writer finishes the CSV record before a backup is captured.. m20-data Pod, Single quiescent writer, active. PVC m20, Mounted /data, idle. Private host backup, Outside cluster, idle. Logical deletion, Removes orders.csv, idlem20-data Pod: Single quiescent writer
m20-data PodSingle quiescent writerIn focus
PVC m20: Mounted /data
PVC m20Mounted /dataContext
Private host backup: Outside cluster
Private host backupOutside clusterContext
Logical deletion: Removes orders.csv
Logical deletionRemoves orders.csvContext
m20-data PodSingle quiescent writerIn focus
PVC m20Mounted /dataContext
Private host backupOutside clusterContext
Logical deletionRemoves orders.csvContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
No retained copy
Write then quiesce

The single writer finishes the CSV record before a backup is captured.

Read the complete visual sequence
  1. Write then quiesce

    The single writer finishes the CSV record before a backup is captured.

  2. Keep an external copy

    The script captures the bounded file in a private host directory and verifies its exact bytes before deleting the live file.

  3. Delete the actual record

    The script removes the file from the mounted PVC and verifies absence; a Pod restart cannot undo this deletion.

No retained copy

Restarting a consumer cannot recover bytes no longer present in its volume.

  1. File is removed

    The current persistent filesystem no longer contains orders.csv.

  2. Pod is recreated

    The new Pod mounts the same claim and sees the same missing file.

  3. Recovery source is absent

    Without a retained backup or application recovery history, persistence alone provides no earlier record.

About this model

The CSV is quiescent during copy; this does not model concurrent database transaction consistency.

Walkthrough: consistency is an explicit assumption

The single writer completes its write before backup, and no concurrent writer changes the file during capture. That quiescent-file assumption makes a byte copy meaningful for this example. A database with concurrent transactions, multiple files, write-ahead logs and caches needs its supported backup or quiescing procedure. A storage snapshot may be crash-consistent without being application-consistent; the required guarantees depend on the application and snapshot implementation. Do not describe this CSV copy as a database backup.

Read capstone-stateful.sh. It writes order-42,confirmed, verifies the captured backup bytes against that fixture before deleting the in-cluster file, confirms absence, then copies the backup into the mounted volume with kubectl cp. It compares exact restored content before and after Pod replacement, reporting byte counts and SHA-256 digests. The PVC remains while the consumer Pod is deleted and recreated from data-consumer.yaml. The data survives because the new Pod mounts the same claim, not because Kubernetes knows how to regenerate the order record.

Restore bytes, then prove a new Pod sees them

A new Pod UID mounts the same PVC and reads the restored content, demonstrating persistence after recovery.

Lifecycle viewStep 1 of 3
Restore the recordkubectl cp restores /data/orders.csv in the existing volume; byte counts and SHA-256 digests must match the private host backup.. Host CSV backup, Retained original record, idle. Restore file copy, kubectl cp; tar in image, active. Existing PVC, Same bound storage, idle. Replacement Pod, New UID, same claim, idle. Content assertion, order-42,confirmed, idleHost CSV backup: Retained original record
Host CSV backupRetained original recordContext
Restore file copy: kubectl cp; tar in image
Restore file copykubectl cp; tar in imageIn focus
Existing PVC: Same bound storage
Existing PVCSame bound storageContext
Replacement Pod: New UID, same claim
Replacement PodNew UID, same claimContext
Content assertion: order-42,confirmed
Content assertionorder-42,confirmedContext
Host CSV backupRetained original recordContext
Restore file copykubectl cp; tar in imageIn focus
Existing PVCSame bound storageContext
Replacement PodNew UID, same claimContext
Content assertionorder-42,confirmedContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Backup captured during database writes
Restore the record

kubectl cp restores /data/orders.csv in the existing volume; byte counts and SHA-256 digests must match the private host backup.

Read the complete visual sequence
  1. Restore the record

    kubectl cp restores /data/orders.csv in the existing volume; byte counts and SHA-256 digests must match the private host backup.

  2. Verify before replacement

    The script checks exact expected content so a successful write command alone is insufficient.

  3. Verify after replacement

    A new Pod UID mounts the same PVC and reads the restored content, demonstrating persistence after recovery.

Backup captured during database writes

A byte copy can be nonempty yet fail application consistency requirements.

  1. Concurrent writers remain active

    A hypothetical database changes related files while capture is in progress.

  2. Restore returns bytes

    The files copy successfully, but their relationships may reflect different transaction points.

  3. Integrity is unproven

    Use application-native backup or validated quiescing and verify transaction-level invariants.

About this model

The actual lab has one completed CSV write and tests file restoration, not database recovery or node-loss resilience.

The PVC below requests a small dynamically provisioned volume using the environment’s default StorageClass. ReadWriteOnce describes a supported access mode; it is not an application-level transaction lock or a guarantee of cross-node portability for local storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: m20
  namespace: kv-lab
  labels:
    lab: m20
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 64Mi
# Run in the downloaded m20 directory.
bash capstone-stateful.sh
# Verify the backup is retained on the host and the restored value is mounted.
backup_dir=$(python3 backup-path.py m20)
cat "$backup_dir/orders.csv"
kubectl --context kind-kv-lab -n kv-lab exec m20-data -- cat /data/orders.csv
kubectl --context kind-kv-lab -n kv-lab get pvc m20 -o wide
kubectl --context kind-kv-lab -n kv-lab get pod m20-data -o jsonpath='{.metadata.uid}{"\n"}'

Practice: make each recovery claim testable

Run the m20 local exercise using capstone-stateful.sh. Record five observations: the original record, the host backup, file absence after deliberate deletion, exact restored content, and the same content after a new Pod UID mounts the PVC. The script performs these assertions so a green result has a clear meaning. It does not measure restore throughput, loss under concurrent writes, database consistency or recovery from a failed storage node.

The script restores this bounded fixture with kubectl cp, which requires tar inside the container; the pinned BusyBox image provides it. It compares byte counts and SHA-256 digests before and after Pod replacement. Its private host output defaults to an operating-system temporary directory outside the website assets. Move real backups to durable protected storage with a defined retention policy.

Read the StorageClass, bound PV and reclaim policy before cleanup. A local provisioner can tie volume data to a particular node or container filesystem. Preserving a PVC while restarting its Pod tests a different failure than losing the node that stores its bytes. Deleting the PVC may trigger backend deletion according to reclaim policy and implementation. Keep the backup outside that deletion scope before removing storage resources.

Pitfalls: data integrity needs application knowledge

A nonempty backup can still contain incomplete records. Checksums prove correspondence between bytes being compared, not that those bytes form a valid application state. For a real database, verify catalog integrity, transaction recovery, record counts or business invariants using its supported tools. Define whether restore overwrites existing records, merges them, or requires an empty target. Rehearse credentials and access to encrypted backups before an incident removes the original key service.

An etcd restore can reconstruct the PVC object without reconstructing the volume’s CSV file. Conversely, the host CSV backup cannot recreate RBAC, Service configuration or control-plane trust. Recovery is the assembly of several asset classes under a tested order. Keep the operator or GitOps controller from immediately reapplying conflicting old intent while that assembly is incomplete.

Check your understanding

The CSV is deleted from the PVC, and the Pod is replaced. Why is the CSV still missing until restoration?

The replacement mounts the same persistent filesystem, including its deletion. Persistence preserves current bytes and metadata; it does not keep an automatic historical version of every file. The external backup supplies the earlier record that the restore procedure writes back.

Next, recover declarative ownership itself. The final capstone coordinates GitOps intent, an operator’s generated resources, conflict handling and garbage collection without making two controllers own the same field.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.