Ownership and failure recovery
Repair missing children while refusing unsafe adoption and cleanup.
What you will understand
- Verify child owner references by parent UID.
- Explain repair after deletion and refusal to adopt unrelated objects.
- Choose finalizers only for cleanup obligations not handled by garbage collection.
Builds on: Leases and leader election, Build an idempotent operator
Mental model: ownership must survive name reuse
A controller manages a relationship between a parent intention and dependent resources. In this lab, the generated Deployment and ConfigMap carry ownerReferences naming the WebApp’s UID and identifying it as controller owner. The UID matters because a WebApp can be deleted and recreated with the same name. The replacement is a new parent identity. A controller that adopts any child with the expected name could overwrite a resource belonging to someone else or a previous lifecycle.
The Python operator checks existing child ownership and refuses unrelated objects. This is a useful safety property even in a tiny controller. Labels such as app=m16 help find resources, but labels are mutable selection metadata, not proof of ownership. If an operator reports an adoption refusal, inspect both the current parent UID and the child’s complete ownerReferences before considering any repair. Do not remove the check simply to make a demonstration proceed.
Stable names do not grant adoption
A matching owner permits normal reconciliation; names and labels alone do not.
The operator uses the current WebApp UID when deriving child owner references.
- Current WebApp UID A to Operatordesired parent identity
Read the complete visual sequence
- Read parent identity
The operator uses the current WebApp UID when deriving child owner references.
- Inspect existing child
A child with the expected name must already have the matching controller owner before it is managed.
- Apply only a valid relationship
A matching owner permits normal reconciliation; names and labels alone do not.
Unrelated object blocks adoption
An existing child belongs to another lifecycle.
- Same name is found
The expected child name is occupied by a resource whose owner UID is B.
- Identity mismatch is detected
The current WebApp UID is A, so the operator refuses to apply over that child.
- Investigate before takeover
Resolve the intended ownership explicitly; removing the safety check risks overwriting another resource.
About this model
The teaching controller intentionally refuses adoption; production adoption policies require additional explicit design.
Walkthrough: repair and deletion are different reconciliations
When the ConfigMap is accidentally deleted while the WebApp still exists, the next pass computes the same desired child and recreates it. Its new UID proves replacement; the parent’s UID remains unchanged. When a Deployment is deleted, the next pass recreates that child too, but application availability may be disrupted while Pods return. Reconciliation provides a route back toward desired state, not instantaneous immunity from destructive changes.
Deleting the parent is different. The teaching operator skips resources with deletionTimestamp and relies on Kubernetes garbage collection for its Kubernetes children. No external system is provisioned, so there is no external cleanup finalizer in this example. A production operator managing a cloud database might need a finalizer to coordinate retention or deletion before the parent disappears. Such cleanup must be idempotent and must handle the external resource already being absent. A finalizer cannot turn a nontransactional pair of systems into one atomic transaction.
Repair a child, then delete the parent
When the parent is removed, garbage collection removes owned children; the operator does not recreate a deleted parent.
With the parent still active, a missing child is drift from desired state.
- WebApp to ConfigMap / DeploymentownerReference UID
Read the complete visual sequence
- Accidental child deletion
With the parent still active, a missing child is drift from desired state.
- Reconcile repairs drift
The next operator pass creates the same desired child name with a new child UID and the same parent UID.
- Parent deletion changes intent
When the parent is removed, garbage collection removes owned children; the operator does not recreate a deleted parent.
External cleanup is required
Kubernetes garbage collection cannot perform a database provider API workflow.
- External resource exists
A hypothetical production extension manages infrastructure outside the API owner graph.
- No finalizer coordination
Deleting the parent without an implemented cleanup barrier can abandon that external resource.
- Implement the obligation
Use an idempotent finalizer protocol with external identity checks; a string alone supplies no cleanup code.
About this model
The actual lab creates only Kubernetes children and needs no external-resource finalizer.
This manifest illustrates the parent relationship through the real WebApp API. Owner references on the children are generated by the controller; do not copy a UID from documentation into a hand-written dependent manifest.
apiVersion: labs.kv.example/v1alpha1
kind: WebApp
metadata:
name: m16
namespace: kv-lab
labels:
lab: m16
spec:
replicas: 1
message: repaired-by-reconciliation
kubectl --context kind-kv-lab -n kv-lab get webapp m16 -o jsonpath='{.metadata.uid}{"\n"}'
kubectl --context kind-kv-lab -n kv-lab get configmap m16 -o jsonpath='{.metadata.ownerReferences}{"\n"}'
# Delete only this generated, disposable ConfigMap.
kubectl --context kind-kv-lab -n kv-lab delete configmap m16
python3 operator.py --once
kubectl --context kind-kv-lab -n kv-lab get configmap m16 -o yaml
Practice: collect evidence at each boundary
Use the m16 local exercise. Record the parent UID, child UID and child owner UID before failure. Delete the generated ConfigMap, run one reconcile pass, and record them again. Expect the child UID to change, its owner UID to match the unchanged WebApp UID, and the page content to match current spec.message. This proves repair toward current intention rather than restoration of a particular old object identity.
Parent deletion is demonstrated again in the m20 platform capstone. Before trying it manually, save the desired manifest and ensure the objects contain no data you need to preserve. Delete the WebApp, wait for dependent removal, and inspect events if deletion stalls. Background garbage collection is asynchronous. A child can remain briefly visible after the parent’s delete request returns. Do not immediately declare ownership broken because two terminal reads caught different points in the deletion sequence.
Pitfalls: cleanup policy belongs in the API design
A database operator needs a documented retention decision: should deleting the custom resource preserve data, require a separate approval, or remove storage? ownerReferences alone cannot express every data policy. A broken finalizer controller can block deletion, while a missing finalizer can leak an external resource. Repeated cleanup attempts must not delete a newly recreated external object with the old name; carry and verify external identity as well as Kubernetes UID. Conditions should identify the failed cleanup step without placing credentials or sensitive data in status messages.
For recovery, pause destructive automation when necessary and understand what will reconcile next. Restoring an old custom resource into a live environment can cause the operator to enact old intent immediately. A backup procedure must include a plan for controller startup and ownership reconciliation, not just an API import.
Check your understanding
A child has the expected name and labels, but its owner UID differs. Should the operator overwrite it?
No. Those facts do not establish that it belongs to the current parent. The lab operator refuses adoption so an operator or user must investigate identity and choose an explicit recovery action rather than silently seizing the child.
Next, relate reconciliation pressure to resource pressure. A reliable loop still depends on enough CPU, memory, API capacity and storage to finish its work.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise