Appearance

Finalizers, garbage collection, and API evolution

Distinguish cleanup barriers, dependent ownership, and versioned representations.

What you will understand

  • Explain why a finalizer can hold deletion indefinitely.
  • Trace dependent deletion using owner UID and propagation policy.
  • Plan API migration without confusing served versions with storage.

Builds on: Watches and server-side apply

Mental model: deletion is a protocol

A delete request does not always immediately remove an object. If finalizers are present, the API server records a deletionTimestamp and retains the object while responsible controllers perform cleanup. A finalizer is a named string, not executable code. Adding labs.kv.example/hold does not create a controller capable of interpreting it. Without a working implementation that removes the string after cleanup, the object can remain terminating indefinitely.

A controller should register its finalizer before creating an external resource whose cleanup must be coordinated. During termination it performs retryable, idempotent cleanup, then removes its own finalizer. Removing someone else’s finalizer manually can abandon a cloud resource, release a safety barrier too early, or lose data. Investigate the controller, its permissions, dependency health and cleanup evidence before treating force removal as repair.

A named barrier requires a controller

After cleanup evidence, the controller removes its own finalizer and deletion can complete.

Lifecycle viewStep 1 of 3
Object enters terminationThe API records deletionTimestamp while preserving an object that still has a finalizer.. Delete request, Requests removal, idle. ConfigMap, Finalizer + deletionTimestamp, active. Cleanup controller, Performs cleanup if implemented, idle. External resource, Cleanup obligation, idleDelete request: Requests removal
Delete requestRequests removalContext
ConfigMap: Finalizer + deletionTimestamp
ConfigMapFinalizer + deletionTimestampIn focus
Cleanup controller: Performs cleanup if implemented
Cleanup controllerPerforms cleanup if implementedContext
External resource: Cleanup obligation
External resourceCleanup obligationContext
Delete requestRequests removalContext
ConfigMapFinalizer + deletionTimestampIn focus
Cleanup controllerPerforms cleanup if implementedContext
External resourceCleanup obligationContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
No implementation for the key
Object enters termination

The API records deletionTimestamp while preserving an object that still has a finalizer.

Read the complete visual sequence
  1. Object enters termination

    The API records deletionTimestamp while preserving an object that still has a finalizer.

  2. Controller fulfills obligation

    The responsible controller cleans its external resource; the finalizer string itself runs no code.

  3. Barrier is removed

    After cleanup evidence, the controller removes its own finalizer and deletion can complete.

No implementation for the key

An artificial training finalizer holds the ConfigMap indefinitely.

  1. Training key is present

    The object has labs.kv.example/hold, but no controller implements that key.

  2. Delete remains pending

    deletionTimestamp is present and the object remains retrievable because its finalizer list is nonempty.

  3. Inspect before intervening

    Only the isolated training key is removed manually; real cleanup obligations require investigation.

About this model

An external cleanup obligation illustrates the mechanism; the lab ConfigMap deliberately has none.

Walkthrough: ownership identifies the dependent, not a name match

Garbage collection uses ownerReferences to connect a dependent to an owner’s UID. A replacement object with the same name has a different identity and is not automatically the old dependent’s owner. A Deployment normally owns ReplicaSets, which own Pods. Namespaced owners and dependents have scope rules; a namespaced owner cannot own a dependent in a different namespace. Labels provide selection and grouping, but a matching label alone does not create a garbage-collection relationship.

Deletion propagation changes what the API caller waits for. Background propagation can remove the owner first while garbage collection proceeds. Foreground propagation keeps a deleting owner present while qualifying blocking dependents are removed; behavior depends on owner references and controller observations. Orphan propagation preserves dependents by removing the relevant ownership relationship. Choose deliberately when data or external infrastructure is involved. An owner reference does not execute database backups before a PVC disappears.

Garbage collection follows owner identity

Eligible ReplicaSets and Pods are removed through their owner graph; unrelated objects remain.

Ownership viewStep 1 of 3
Record actual ownerThe ReplicaSet references Deployment UID A; labels do not define this lifecycle edge.. Deployment UID A, Owner of ReplicaSet, idle. ReplicaSet, ownerReference UID A, active. Pod, Owned by ReplicaSet, idle. Garbage collector, Traverses owner references, idle. Deployment UID B, Same name, new identity, idleDeployment UID A: Owner of ReplicaSet
Deployment UID AOwner of ReplicaSetContext
ReplicaSet: ownerReference UID A
ReplicaSetownerReference UID AIn focus
Pod: Owned by ReplicaSet
PodOwned by ReplicaSetContext
Garbage collector: Traverses owner references
Garbage collectorTraverses owner referencesContext
Deployment UID B: Same name, new identity
Deployment UID BSame name, new identityContext
Deployment UID AOwner of ReplicaSetContext
ReplicaSetownerReference UID AIn focus
PodOwned by ReplicaSetContext
Garbage collectorTraverses owner referencesContext
Deployment UID BSame name, new identityContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Name reuse is not adoption
Record actual owner

The ReplicaSet references Deployment UID A; labels do not define this lifecycle edge.

Read the complete visual sequence
  1. Record actual owner

    The ReplicaSet references Deployment UID A; labels do not define this lifecycle edge.

  2. Choose propagation

    Deleting UID A initiates background, foreground or orphan behavior according to the selected policy.

  3. Collect dependent chain

    Eligible ReplicaSets and Pods are removed through their owner graph; unrelated objects remain.

Name reuse is not adoption

A new Deployment with UID B does not become the old dependent owner.

  1. Owner A is gone

    The dependent still names owner UID A even after the original name becomes available.

  2. Replacement appears

    A Deployment reuses the name but receives UID B. The reference remains tied to UID A.

  3. No implicit repair

    Garbage collection and controllers do not treat equal names as equal identities. Investigate intended adoption.

About this model

The diagram summarizes garbage collection; foreground blocking and cache timing affect the exact removal sequence.

API evolution is another lifecycle protocol. A served version is a representation clients can request; storage version is how objects are persisted. Deprecating or removing an API endpoint does not automatically update Git manifests, controllers, admission webhook matching rules or stored custom-resource versions. Discover consumers, migrate requests, test conversion and update stored objects before disabling an old representation. Kubernetes’ deprecation policy distinguishes stability levels: GA API versions may be deprecated but cannot be removed within a Kubernetes major version; beta and alpha versions have different removal rules. Check the exact resource and release path, and define a separate version policy for your own CRDs.

Migrate consumers before removing an API

Check stored-version migration and release policy before disabling the old representation.

Flow viewStep 1 of 3
Inventory consumersIdentify controllers, manifests and webhook rules using the deprecated endpoint before an upgrade.. Old API client, Uses deprecated version, active. API server, Served representations, idle. Stored objects, Storage representation, idle. Updated client, Uses supported version, idleOld API client: Uses deprecated version
Old API clientUses deprecated versionIn focus
API server: Served representations
API serverServed representationsContext
Stored objects: Storage representation
Stored objectsStorage representationContext
Updated client: Uses supported version
Updated clientUses supported versionContext
Old API clientUses deprecated versionIn focus
API serverServed representationsContext
Stored objectsStorage representationContext
Updated clientUses supported versionContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Endpoint removed too early
Inventory consumers

Identify controllers, manifests and webhook rules using the deprecated endpoint before an upgrade.

Read the complete visual sequence
  1. Inventory consumers

    Identify controllers, manifests and webhook rules using the deprecated endpoint before an upgrade.

  2. Migrate representation

    Update clients and test conversion while old and new supported versions are still available.

  3. Verify storage and removal path

    Check stored-version migration and release policy before disabling the old representation.

Endpoint removed too early

Old clients fail even when equivalent objects still exist in storage.

  1. Server no longer serves old version

    The upgrade removes the endpoint expected by the old client.

  2. Client request fails

    An old manifest or controller cannot use its requested representation.

  3. Stored object is not a client upgrade

    Existing persisted state does not rewrite Git manifests or update client libraries.

About this model

Specific removal releases and CRD conversion mechanics must be checked for the chosen resource and upgrade path.

This isolated ConfigMap demonstrates a hold with no cleanup code. Its finalizer is intentionally artificial, so removing it after observation is the completion of this exercise, not a production recovery recommendation.

apiVersion: v1
kind: ConfigMap
metadata:
  name: m14-held
  namespace: kv-lab
  labels:
    lab: m14
  finalizers:
    - labs.kv.example/hold # No controller implements this training key.
data:
  purpose: inspect-deletion
kubectl --context kind-kv-lab -n kv-lab apply -f held.yaml
kubectl --context kind-kv-lab -n kv-lab delete configmap m14-held --wait=false
kubectl --context kind-kv-lab -n kv-lab get configmap m14-held -o yaml
# Only the artificial hold is removed in this isolated exercise.
kubectl --context kind-kv-lab -n kv-lab patch configmap m14-held --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl --context kind-kv-lab -n kv-lab wait --for=delete configmap/m14-held --timeout=30s

Practice: prove which lifecycle is active

Run the m14 local exercise, which includes a similar explicit hold. Expect deletionTimestamp to appear while the ConfigMap can still be fetched. Compare this with a ReplicaSet owner reference from an earlier Deployment lab. Record the owner’s kind, namespace, name and UID, then compare that UID with the current owner. These observations distinguish a deletion barrier from a dependent relationship and from an unrelated label.

For migration practice, inspect kubectl api-resources and the apiVersion in your committed manifest. Discuss a migration checklist before changing a cluster version: request compatibility, webhook compatibility, storage conversion, controller behavior and rollback boundaries. The local finalizer demonstration does not validate all of those contracts.

Pitfalls: a shorter object list can hide damage

Force removal can make a terminating object vanish while its external obligations remain. Deleting a CRD can remove all objects of that custom type, which is far more consequential than stopping an operator process. A status field saying cleanup succeeded is only meaningful if the controller correctly owns and updates it. Keep durable cleanup evidence for destructive operations and avoid making safety depend on a single ephemeral log line.

Check your understanding

A new parent has the same name as the deleted parent. Does it own the old child?

No. Garbage collection follows owner identity, including UID, rather than names alone. The replacement parent’s UID differs. Inspect and intentionally repair ownership only after establishing which controller and resource should be responsible.

Next, build a controller that treats repeated observations and partial cleanup as expected operating conditions. Deletion correctness requires the same reconciliation discipline as creation.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.