Appearance

Desired state and reconciliation

See how repeated observation and repair turn intent into eventual convergence.

What you will understand

  • Read desired and observed replica counts separately
  • Explain idempotent repeated reconciliation
  • Predict recovery while keeping component ownership precise

Builds on: Inside a Kubernetes cluster

Mental model: compare, act, observe again

A reconciliation loop repeatedly compares a requested state with observed state and takes bounded actions to reduce the difference. You request three replicas, not a one-time instruction to “start three processes immediately.” If one managed Pod disappears later, the requested count still exists. A controller can notice the difference and create a replacement object without another command from you.

Desired state and observed state are separate records with separate timing. spec.replicas expresses an intention. Status fields report observations such as how many replicas exist, are ready, or are available. A watch notification is a signal to reconsider the latest state; it is not permission to blindly repeat every previous action. Kubernetes controllers must tolerate repeated work, missed intermediate changes, and delays. An idempotent action has the same intended effect when repeated against the same relevant state.

For replica-count repair, a ReplicaSet counts active managed Pods. An unready or image-pull-blocked Pod can still count as active; a terminal Failed or Succeeded Pod, or a Pod already marked for deletion, does not fill that target. Retained terminal objects can therefore appear alongside replacements in a listing. Count the relevant lifecycle state, not every row returned by get pods.

“Eventually consistent” does not mean “will certainly become healthy.” Convergence requires a feasible request and functioning dependencies. A controller can create the requested active Pods while every container fails to pull the same nonexistent image. The kubelets retry image preparation; the ReplicaSet does not replace a still-active Pod merely because it is waiting for an image. The loops can be working as designed while the workload remains unavailable. Always ask which difference is being repaired and whether the environment permits that repair.

Close the observed-state gap

A ReplicaSet controller compares its target with active owned Pods.

Ownership viewStep 1 of 3
Declare three replicasThe ReplicaSet target is three but only two matching active owned Pods exist. Terminal or deleting Pods do not fill the target. The controller observes a count gap.. ReplicaSet desired: 3, active. ReplicaSet controller, active. Pod A, ready. Pod B, readyReplicaSet desired: 3
ReplicaSet desired: 3In focus
ReplicaSet controller
ReplicaSet controllerIn focus
Pod A
Pod AReady
Pod B
Pod BReady
ReplicaSet desired: 3In focus
ReplicaSet controllerIn focus
Pod AReady
Pod BReady
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Failure: the target is infeasible
Declare three replicas

The ReplicaSet target is three but only two matching active owned Pods exist. Terminal or deleting Pods do not fill the target. The controller observes a count gap.

Read the complete visual sequence
  1. Declare three replicas

    The ReplicaSet target is three but only two matching active owned Pods exist. Terminal or deleting Pods do not fill the target. The controller observes a count gap.

  2. Create one missing Pod

    The ReplicaSet controller creates one Pod object. It does not create three more because three was the original request.

  3. Observe again before acting

    Three active managed Pods now exist. A repeated reconciliation observes no count gap and does not duplicate them. Readiness is still a separate observation.

Failure: the target is infeasible

Failure: the target is infeasible

  1. Replica target remains three

    The desired count is valid even though the referenced image is missing from its registry.

  2. Objects exist without health

    The controller can satisfy the object count while containers fail image pulls. Count convergence does not prove readiness.

  3. Correct the enduring input

    Fix the owning Deployment image reference. Repeatedly deleting these Pods would reproduce the same bad template.

About this model

A Deployment has already set its ReplicaSet target to three; all observed Pods are healthy in this branch.

Walkthrough: write an enduring intention

This example creates a Deployment so its controller manages a ReplicaSet, whose controller creates Pods. Deployment details come later; here the ownership chain makes repeated repair observable. Save it as reconcile-demo.yaml and use a disposable lab namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reconcile-demo
spec:
  replicas: 3 # Enduring desired Pod count.
  selector:
    matchLabels:
      app: reconcile-demo
  template:
    metadata:
      labels:
        app: reconcile-demo # Must match the selector.
    spec:
      containers:
        - name: web
          image: nginx:1.27.5
kubectl apply -f reconcile-demo.yaml
kubectl get deployment reconcile-demo
kubectl get rs -l app=reconcile-demo
kubectl get pods -l app=reconcile-demo -w

Wait for three healthy Pods if your lab has enough resources and registry access. In another terminal, list their names and delete exactly one named Pod from this disposable workload. Do not delete a node or a system Pod to imitate a failure.

kubectl get pods -l app=reconcile-demo
# Replace the example name with one actual Pod name from that listing.
kubectl delete pod ACTUAL_POD_NAME
kubectl get pods -l app=reconcile-demo -o wide

The ReplicaSet controller creates a replacement Pod because the desired count remains three. The scheduler binds that new Pod to an eligible node. The node’s kubelet and runtime start it. The original Pod is not revived with its original UID or migrated to a new node. Names and IPs can change, and local ephemeral state follows its own lifecycle.

Replacement is a new identity

Trace deletion, replacement creation, binding, and execution.

Lifecycle viewStep 1 of 3
A managed Pod disappearsDeleting Pod B leaves the ReplicaSet target unchanged. Remaining Pods continue independently.. ReplicaSet desired: 3, active. Pod B: deleted UID, failed. Pods A and C, readyReplicaSet desired: 3
ReplicaSet desired: 3In focus
Pod B: deleted UID
Pod B: deleted UIDFailed
Pods A and C
Pods A and CReady
ReplicaSet desired: 3In focus
Pod B: deleted UIDFailed
Pods A and CReady
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Failure: no controller owns the deleted Pod
A managed Pod disappears

Deleting Pod B leaves the ReplicaSet target unchanged. Remaining Pods continue independently.

Read the complete visual sequence
  1. A managed Pod disappears

    Deleting Pod B leaves the ReplicaSet target unchanged. Remaining Pods continue independently.

  2. The ReplicaSet creates Pod D

    The ReplicaSet controller creates a new object with a new UID. Scheduling assigns that object to a node.

  3. The worker starts D

    The selected node kubelet uses its runtime to start D. Recovery preserves replica intent, not B’s process memory or Pod IP.

Failure: no controller owns the deleted Pod

Failure: no controller owns the deleted Pod

  1. A naked Pod exists

    A directly created Pod has no ReplicaSet maintaining a count for it.

  2. Delete the only intention

    Deleting that Pod removes the object. There is no replica controller with an outstanding count gap.

  3. Use a managed workload

    For a replaceable long-running application, declare a Deployment. Its ReplicaSet provides the enduring replacement obligation.

About this model

One Pod is deleted from a healthy three-replica workload; termination overlap is simplified.

Practice: repair intention versus fight it

Scale the Deployment to two replicas with kubectl scale deployment reconcile-demo --replicas=2. The observed count should converge toward two. Now reapply the original file, which still declares three. The file again requests three, and the controller works toward that count. This is configuration drift in a small form: multiple writers disagree about the same desired field. Keep an authoritative manifest or a deliberate field-ownership workflow instead of treating every manual edit as a permanent new truth.

Observe the workload while it converges. A new Pod can exist before it is ready, and an old terminating Pod can remain visible briefly. Exact names, event ordering, and timings vary. Focus on enduring invariants: the requested count, owner relationships, whether new Pods are created, whether they bind, and whether they become usable. Reconciliation is asynchronous, so one instantaneous listing is not proof of a stable final state.

Pitfalls

Deleting a naked Pod removes its intention as well as its execution object; without a controller, no ReplicaSet is obligated to replace it. Editing a managed Pod directly is also a weak release strategy: changes to a replacement come from the controller’s template. Change the owning workload’s desired configuration.

Do not confuse process restart with replica replacement. A kubelet can restart a failed container in an existing Pod. A ReplicaSet controller repairs a missing Pod by creating another Pod object. Those repairs occur at different layers and preserve different identities.

Check your understanding

Three Pods exist, but none are ready. Has reconciliation achieved the user’s goal?

The replica-count portion may be satisfied while application availability remains unsatisfied. Readiness, image failures, configuration, resource constraints, and dependencies determine whether those replicas serve useful work. Inspect both count and health. Increasing the requested count can create more failing replicas without solving the cause.

Next, trace one Pod request through every responsible component. Practice deletion and scaling in the module lab, then remove the demo Deployment to release lab resources.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.