Desired state and reconciliation
See how repeated observation and repair turn intent into eventual convergence.
What you will understand
- Read desired and observed replica counts separately
- Explain idempotent repeated reconciliation
- Predict recovery while keeping component ownership precise
Builds on: Inside a Kubernetes cluster
Mental model: compare, act, observe again
A reconciliation loop repeatedly compares a requested state with observed state and takes bounded actions to reduce the difference. You request three replicas, not a one-time instruction to “start three processes immediately.” If one managed Pod disappears later, the requested count still exists. A controller can notice the difference and create a replacement object without another command from you.
Desired state and observed state are separate records with separate timing. spec.replicas expresses an intention. Status fields report observations such as how many replicas exist, are ready, or are available. A watch notification is a signal to reconsider the latest state; it is not permission to blindly repeat every previous action. Kubernetes controllers must tolerate repeated work, missed intermediate changes, and delays. An idempotent action has the same intended effect when repeated against the same relevant state.
For replica-count repair, a ReplicaSet counts active managed Pods. An unready or image-pull-blocked Pod can still count as active; a terminal Failed or Succeeded Pod, or a Pod already marked for deletion, does not fill that target. Retained terminal objects can therefore appear alongside replacements in a listing. Count the relevant lifecycle state, not every row returned by get pods.
“Eventually consistent” does not mean “will certainly become healthy.” Convergence requires a feasible request and functioning dependencies. A controller can create the requested active Pods while every container fails to pull the same nonexistent image. The kubelets retry image preparation; the ReplicaSet does not replace a still-active Pod merely because it is waiting for an image. The loops can be working as designed while the workload remains unavailable. Always ask which difference is being repaired and whether the environment permits that repair.
Close the observed-state gap
A ReplicaSet controller compares its target with active owned Pods.
The ReplicaSet target is three but only two matching active owned Pods exist. Terminal or deleting Pods do not fill the target. The controller observes a count gap.
- ReplicaSet controller to ReplicaSet desired: 3read target
- ReplicaSet desired: 3 to Pod Aowns
- ReplicaSet desired: 3 to Pod Bowns
Read the complete visual sequence
- Declare three replicas
The ReplicaSet target is three but only two matching active owned Pods exist. Terminal or deleting Pods do not fill the target. The controller observes a count gap.
- Create one missing Pod
The ReplicaSet controller creates one Pod object. It does not create three more because three was the original request.
- Observe again before acting
Three active managed Pods now exist. A repeated reconciliation observes no count gap and does not duplicate them. Readiness is still a separate observation.
Failure: the target is infeasible
Failure: the target is infeasible
- Replica target remains three
The desired count is valid even though the referenced image is missing from its registry.
- Objects exist without health
The controller can satisfy the object count while containers fail image pulls. Count convergence does not prove readiness.
- Correct the enduring input
Fix the owning Deployment image reference. Repeatedly deleting these Pods would reproduce the same bad template.
About this model
A Deployment has already set its ReplicaSet target to three; all observed Pods are healthy in this branch.
Walkthrough: write an enduring intention
This example creates a Deployment so its controller manages a ReplicaSet, whose controller creates Pods. Deployment details come later; here the ownership chain makes repeated repair observable. Save it as reconcile-demo.yaml and use a disposable lab namespace.
apiVersion: apps/v1
kind: Deployment
metadata:
name: reconcile-demo
spec:
replicas: 3 # Enduring desired Pod count.
selector:
matchLabels:
app: reconcile-demo
template:
metadata:
labels:
app: reconcile-demo # Must match the selector.
spec:
containers:
- name: web
image: nginx:1.27.5
kubectl apply -f reconcile-demo.yaml
kubectl get deployment reconcile-demo
kubectl get rs -l app=reconcile-demo
kubectl get pods -l app=reconcile-demo -w
Wait for three healthy Pods if your lab has enough resources and registry access. In another terminal, list their names and delete exactly one named Pod from this disposable workload. Do not delete a node or a system Pod to imitate a failure.
kubectl get pods -l app=reconcile-demo
# Replace the example name with one actual Pod name from that listing.
kubectl delete pod ACTUAL_POD_NAME
kubectl get pods -l app=reconcile-demo -o wide
The ReplicaSet controller creates a replacement Pod because the desired count remains three. The scheduler binds that new Pod to an eligible node. The node’s kubelet and runtime start it. The original Pod is not revived with its original UID or migrated to a new node. Names and IPs can change, and local ephemeral state follows its own lifecycle.
Replacement is a new identity
Trace deletion, replacement creation, binding, and execution.
Deleting Pod B leaves the ReplicaSet target unchanged. Remaining Pods continue independently.
- ReplicaSet desired: 3 to Pods A and Cowns
Read the complete visual sequence
- A managed Pod disappears
Deleting Pod B leaves the ReplicaSet target unchanged. Remaining Pods continue independently.
- The ReplicaSet creates Pod D
The ReplicaSet controller creates a new object with a new UID. Scheduling assigns that object to a node.
- The worker starts D
The selected node kubelet uses its runtime to start D. Recovery preserves replica intent, not B’s process memory or Pod IP.
Failure: no controller owns the deleted Pod
Failure: no controller owns the deleted Pod
- A naked Pod exists
A directly created Pod has no ReplicaSet maintaining a count for it.
- Delete the only intention
Deleting that Pod removes the object. There is no replica controller with an outstanding count gap.
- Use a managed workload
For a replaceable long-running application, declare a Deployment. Its ReplicaSet provides the enduring replacement obligation.
About this model
One Pod is deleted from a healthy three-replica workload; termination overlap is simplified.
Practice: repair intention versus fight it
Scale the Deployment to two replicas with kubectl scale deployment reconcile-demo --replicas=2. The observed count should converge toward two. Now reapply the original file, which still declares three. The file again requests three, and the controller works toward that count. This is configuration drift in a small form: multiple writers disagree about the same desired field. Keep an authoritative manifest or a deliberate field-ownership workflow instead of treating every manual edit as a permanent new truth.
Observe the workload while it converges. A new Pod can exist before it is ready, and an old terminating Pod can remain visible briefly. Exact names, event ordering, and timings vary. Focus on enduring invariants: the requested count, owner relationships, whether new Pods are created, whether they bind, and whether they become usable. Reconciliation is asynchronous, so one instantaneous listing is not proof of a stable final state.
Pitfalls
Deleting a naked Pod removes its intention as well as its execution object; without a controller, no ReplicaSet is obligated to replace it. Editing a managed Pod directly is also a weak release strategy: changes to a replacement come from the controller’s template. Change the owning workload’s desired configuration.
Do not confuse process restart with replica replacement. A kubelet can restart a failed container in an existing Pod. A ReplicaSet controller repairs a missing Pod by creating another Pod object. Those repairs occur at different layers and preserve different identities.
Check your understanding
Three Pods exist, but none are ready. Has reconciliation achieved the user’s goal?
The replica-count portion may be satisfied while application availability remains unsatisfied. Readiness, image failures, configuration, resource constraints, and dependencies determine whether those replicas serve useful work. Inspect both count and health. Increasing the requested count can create more failing replicas without solving the cause.
Next, trace one Pod request through every responsible component. Practice deletion and scaling in the module lab, then remove the demo Deployment to release lab resources.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise