Appearance

Watches and server-side apply

Follow a resumable watch cursor and negotiate field ownership explicitly.

What you will understand

  • Resume watches without treating resourceVersion as a clock.
  • Explain stale caches and relisting after expired history.
  • Resolve server-side apply conflicts through an ownership decision.

Builds on: Request lifecycle and admission

Mental model: observation and ownership solve different problems

A watch delivers changes to a selected API collection. A client typically lists a baseline, then watches from its resourceVersion, preserving the string unchanged. Current kube-apiserver resources, including CRDs, support ordering versions within the same API group and resource type using arbitrary-size decimal comparison; conformant implementations guarantee this from Kubernetes 1.35. Extension APIs require checking their contract and numeric format. This permits constrained ordering, not subtraction to count changes, cross-resource ordering, or elapsed-time calculations.

A disconnected client has not necessarily lost its ability to recover. It can reconnect from its last accepted version while history remains available. If the requested history is too old, the server can return an expired-history response such as HTTP 410 Gone. The client must rebuild its baseline and continue watching. A watch is therefore a resumable observation protocol, not a permanent audit log or a guarantee that every intermediate state reaches every controller.

List, resume, and rebuild a watch

A worker reads the cache and compares current observations with desired state. Cache visibility can lag an API write.

Flow viewStep 1 of 3
Establish the baselineA list supplies collection state and a resourceVersion cursor. Preserve the string unchanged and follow the documented same-resource ordering contract.. API server, Collection and watch history, idle. List baseline, Objects + resumable cursor, active. Watch stream, Changes after cursor, idle. Client cache, Eventually updated local state, idleAPI server: Collection and watch history
API serverCollection and watch historyContext
List baseline: Objects + resumable cursor
List baselineObjects + resumable cursorIn focus
Watch stream: Changes after cursor
Watch streamChanges after cursorContext
Client cache: Eventually updated local state
Client cacheEventually updated local stateContext
API serverCollection and watch historyContext
List baselineObjects + resumable cursorIn focus
Watch streamChanges after cursorContext
Client cacheEventually updated local stateContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
History expired
Establish the baseline

A list supplies collection state and a resourceVersion cursor. Preserve the string unchanged and follow the documented same-resource ordering contract.

Read the complete visual sequence
  1. Establish the baseline

    A list supplies collection state and a resourceVersion cursor. Preserve the string unchanged and follow the documented same-resource ordering contract.

  2. Observe changes

    The client watches from the cursor and updates local state as events arrive; this is not a durable audit log.

  3. Reconcile cached state

    A worker reads the cache and compares current observations with desired state. Cache visibility can lag an API write.

History expired

A disconnected client cannot assume old watch history is retained forever.

  1. Reconnect from old cursor

    The client resumes with its last accepted token after a long interruption.

  2. 410 Gone

    The server cannot provide that retained history; the client discards its incomplete baseline.

  3. Relist and resume

    A new list rebuilds current state and supplies a fresh cursor before the watch resumes.

About this model

A traditional list-then-watch client is shown; optional streaming-list optimizations are omitted. Documented same-resource version ordering does not make the cursor a clock or event count.

Walkthrough: watch current state, negotiate field changes

An informer uses watch information to maintain a local cache, but cache reads can lag a successful write. A reconcile loop must handle an object that is already changed on the API server while its cached copy still looks old. Kubernetes controllers work toward current desired state rather than executing an irreversible business transaction once for every watch notification. For durable audit requirements, configure an audit pipeline with its own retention and access model instead of retaining a controller’s event stream.

Server-side apply solves a separate coordination problem: multiple actors declare intentions for fields. The API server records field managers in managedFields. When an apply changes a field owned by another manager to a different value, it normally reports a conflict. Managers may share ownership when applying the same value; an ordinary update or patch can also change field ownership according to its operation. Ownership is field-specific and schema-dependent, including list merge behavior; it is not an exclusive lock on the entire object.

Two managers and one declared field

A second manager can declare a separate field, or share ownership by applying the same value.

Ownership viewStep 1 of 3
First declarationkv-gitops applies data.release and becomes an owner of that declared field.. kv-gitops, Declares data.release, active. kv-human, Attempts another value, idle. Server-side apply, Computes field ownership, idle. ConfigMap, release: git-declared, idle. managedFields, Records field managers, idlekv-gitops: Declares data.release
kv-gitopsDeclares data.releaseIn focus
kv-human: Attempts another value
kv-humanAttempts another valueContext
Server-side apply: Computes field ownership
Server-side applyComputes field ownershipContext
ConfigMap: release: git-declared
ConfigMaprelease: git-declaredContext
managedFields: Records field managers
managedFieldsRecords field managersContext
kv-gitopsDeclares data.releaseIn focus
kv-humanAttempts another valueContext
Server-side applyComputes field ownershipContext
ConfigMaprelease: git-declaredContext
managedFieldsRecords field managersContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Conflicting declaration
First declaration

kv-gitops applies data.release and becomes an owner of that declared field.

Read the complete visual sequence
  1. First declaration

    kv-gitops applies data.release and becomes an owner of that declared field.

  2. Record the contract

    The stored value and managedFields describe the result; unrelated fields can belong to other managers.

  3. Agree or separate fields

    A second manager can declare a separate field, or share ownership by applying the same value.

Conflicting declaration

A second manager changes an owned field to a different value.

  1. Another intention

    kv-human applies data.release with a value different from the git-managed value.

  2. Conflict is returned

    The API rejects that apply field change; the conflict names the competing field manager.

  3. Make an ownership decision

    The original value remains until actors agree, relinquish the field, or an authorized takeover is explicit.

About this model

One scalar field illustrates ownership; map and list ownership depend on the API schema.

Save this as ownership.yaml in a disposable directory. Manager kv-gitops owns the declared data field after applying it. A second apply using another manager and a different value should conflict.

apiVersion: v1
kind: ConfigMap
metadata:
  name: m14-ownership
  namespace: kv-lab
  labels:
    lab: m14
# Only this key is part of this manager's intention.
data:
  release: git-declared
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-gitops -f ownership.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m14-ownership -o yaml --show-managed-fields
# Keep a second terminal watching while you edit and reapply the file.
kubectl --context kind-kv-lab -n kv-lab get configmap m14-ownership --watch
# After changing data.release locally, expect a field conflict here.
kubectl --context kind-kv-lab -n kv-lab apply --server-side --field-manager=kv-human -f ownership.yaml

Practice: capture a conflict instead of suppressing it

Run the m14 local exercise. It creates a ConfigMap with one field manager and attempts a conflicting declaration with another. Observe the exact field path in the conflict and inspect managedFields before choosing a resolution. For this experiment the intended result is the conflict itself; do not add force simply to make every command green. Watch delivery may group closely timed changes differently from your terminal expectations, so inspect final state as well as the event stream.

There are three useful resolutions to discuss. The second actor can stop declaring the field, both actors can agree on the same value, or an authorized owner can deliberately take ownership using force conflicts. That last option changes the ownership contract. Record who will manage the field on the next reconciliation, otherwise two automated systems can repeatedly undo each other’s changes.

Pitfalls: persistence is not cache synchronization

A successful apply is evidence about the API write, not proof that every cache has observed it. Check the resource’s documented status conditions and, where reported, whether observedGeneration covers the desired generation; metadata.generation alone records desired-state changes, not convergence. A watch bookmark advances progress information without representing a normal object change, and clients must interpret event types correctly. Use maintained client libraries for list/watch recovery. Do not invent a cursor or use resourceVersion: "0" as a universal request for the newest possible read; resourceVersion semantics depend on the operation and associated options.

Check your understanding

A watch cursor changes from one numeric-looking string to another. How many updates occurred?

The strings do not count updates. Documented same-resource ordering does not make resourceVersion a per-object event counter or a clock. Reconcile observed state; use an audit or application log when individual changes matter.

Next, connect field ownership to deletion ownership. managedFields controls declarations of fields; ownerReferences and finalizers control different lifecycle relationships.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.