GitOps and drift reconciliation
Make declared ownership and live recovery agree.
What you will understand
- Trace Git revision through render, diff, sync and health.
- Explain Argo CD self-heal and prune as separate choices.
- Choose an explicit field-ownership takeover for the local drift exercise.
Builds on: Watches and server-side apply, Ownership and failure recovery
Mental model: the source of intent is another dependency
GitOps stores reviewed desired configuration in a source repository and uses a controller to compare that source with live cluster state. Argo CD renders the selected revision, compares desired resources with the destination, and applies changes according to application policy. The controller also evaluates resource health. A successful sync says something about configuration convergence; a healthy application requires workload and dependency evidence. A Git commit is not an atomic transaction across every resource in the cluster.
Drift is a difference between the intended source and observed live state. It can come from an emergency edit, defaulting, an operator, an autoscaler or an unauthorized change. Treating all differences as malicious is as unhelpful as ignoring all differences. Define which actor owns each field. For an HPA-managed workload, continuously forcing the Git replica count can create a fight between controllers rather than a more reliable deployment process.
Revision, render, diff, sync, health
A completed sync does not prove ready endpoints or successful application requests.
The delivery controller obtains source and constructs desired Kubernetes resources.
- Reviewed Git revision to Render manifestsfetch revision
- Render manifests to Compare live statedesired resources
Read the complete visual sequence
- Render a known revision
The delivery controller obtains source and constructs desired Kubernetes resources.
- Compare and act by policy
A difference is evaluated under configured sync, self-heal and prune behavior rather than assumed to trigger every action.
- Evaluate health separately
A completed sync does not prove ready endpoints or successful application requests.
Source cannot render
A source or rendering failure prevents new desired manifests reaching the API.
- Revision is selected
The controller knows which source revision it intends to deploy.
- Render fails
A missing value or invalid source prevents a valid desired manifest set.
- Live resources may keep serving
Existing workloads can remain healthy while delivery is blocked; diagnose source rendering rather than restarting Pods.
About this model
Argo CD is an optional add-on; the mandatory local exercise models the ownership loop with kubectl.
Walkthrough: automatic sync has independent controls
Argo CD automated synchronization, self-healing and pruning are related but distinct settings. Self-healing controls automatic correction of certain live-only deviations under automated sync. Pruning allows removal of tracked resources that are no longer in desired source, with additional safeguards and options relevant to empty applications and destructive changes. Enabling one behavior should not be assumed to enable every other one. Inspect the installed Argo CD version and application policy before predicting whether a manual edit will be reversed.
SSA changes the coordination behavior at the API field boundary. A controller applying a field already owned with a different value can conflict. --force-conflicts is an explicit takeover, not a harmless way to silence a warning. The local m19 exercise first declares a ConfigMap using kv-gitops, introduces drift with an ordinary patch, and then intentionally forces the declarative manager to recover its field. That force is part of the teaching scenario’s approved ownership decision, not a recommended blanket Argo CD policy.
Drift repair includes an ownership decision
The lab intentionally uses force-conflicts for this disposable field, restoring the source value and ownership contract.
kv-gitops applies data.release as git-declared.
- kv-gitops manager to ConfigMap m19initial apply
Read the complete visual sequence
- Declare the source value
kv-gitops applies data.release as git-declared.
- Observe manual drift
A patch changes the field and may transfer ownership; inspect both live value and managedFields.
- Restore with explicit authority
The lab intentionally uses force-conflicts for this disposable field, restoring the source value and ownership contract.
Controllers fight over the same field
Repeated takeovers can produce oscillation instead of convergence.
- Two managers declare replicas
A delivery controller and autoscaler both insist on different values.
- Each recovery undoes the other
Broad forcing repeatedly changes live state and ownership.
- Split responsibilities
Remove the field from one declaration or define a supported collaboration policy rather than forcing indefinitely.
About this model
The drift exercise uses an ordinary patch; conflicting SSA by a second manager is a separate API operation.
This ConfigMap is the entire desired resource in the local exercise. The lab models reconciliation with explicit kubectl commands; it does not silently install an Argo CD service or connect to a remote repository.
apiVersion: v1
kind: ConfigMap
metadata:
name: m19
namespace: kv-lab
labels:
lab: m19
data:
release: git-declared
# Run from the downloaded m19 directory.
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-gitops -f resources.yaml
kubectl --context kind-kv-lab -n kv-lab patch configmap m19 --type=merge -p '{"data":{"release":"manual-drift"}}'
kubectl --context kind-kv-lab -n kv-lab get configmap m19 -o yaml --show-managed-fields
# Intentional takeover of this disposable lab field only.
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-gitops --force-conflicts -f resources.yaml
Practice: observe state and ownership before restoring it
Run the m19 local exercise. Expect release to change to manual-drift, then return to git-declared after explicit declarative recovery. Inspect managedFields before and after the patch because an ordinary update can transfer ownership of changed fields. A separate manager performing a conflicting SSA declaration has different behavior from a patch that already changed the field. These are distinct operations, not interchangeable ways to demonstrate a conflict.
For an optional real Argo CD extension, read ARGOCD-OPTIONAL.md from the download. First choose a supported release and check its upstream-tested Kubernetes versions; a pinned install manifest alone does not establish compatibility with this course’s Kubernetes 1.37 cluster. Keep source credentials, destination permissions, project restrictions and application scope explicit. Test rendering before sync and use an immutable reviewed revision when reproducibility matters. Record both the desired revision and observed health. Repository access failure, manifest-render failure, admission rejection and an unhealthy deployed workload need different diagnosis paths even though they can all prevent a delivery from completing.
Pitfalls: emergency changes need a return path
An emergency live change may be reverted by active self-healing unless the source or reconciliation policy is updated deliberately. Pause the relevant automation only through the team’s incident procedure, preserve the exact live change, and incorporate or revert it in source afterward. Broad ignore-differences rules can hide meaningful drift; broad pruning permissions can delete resources that were incorrectly tracked. Restoring an old cluster snapshot while GitOps points at a newer revision can immediately change the restored state again.
Choose the application boundary to match ownership. GitOps can own the WebApp custom resource while an operator owns generated ConfigMaps and Deployments. Having both controllers declare the same child fields produces avoidable conflicts. This boundary is exercised in the final platform capstone.
Check your understanding
The app is Synced, but no ready endpoints exist. Has GitOps proved service availability?
No. Sync compares and applies desired configuration; readiness and actual requests are later evidence. Inspect application health, Pod conditions, EndpointSlices and dependencies before claiming the service is available.
Next, define who is allowed to supply that desired configuration and which artifacts it may reference. Git history alone is not a complete tenancy or software-trust boundary.
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise