Appearance

Informers, caches, and workqueues

Build convergence from imperfect observations and repeatable work.

What you will understand

  • Trace a shared informer from list and watch to cache handlers.
  • Explain key deduplication and rate-limited reconciliation.
  • Design retries that recover from partial success.

Builds on: Watches and server-side apply

Mental model: events schedule a question, not a command

An informer combines list/watch observation with a local object cache and event handlers. A shared informer lets multiple consumers share that observation machinery. Handlers usually enqueue a resource key such as kv-lab/m16, rather than perform long network calls immediately. A worker later asks: given the desired object and observed dependents now, what must change? This separates API event delivery from expensive reconciliation and keeps the system responsive during bursts.

The cache is eventually consistent with the API server. A worker may successfully create a child and then briefly read a cached parent or child that predates that write. Reconciliation must tolerate that window. It should also tolerate an object disappearing between enqueue and processing. A missing parent often means no creation work is needed; it is not automatically a retryable outage. Do not derive correctness from receiving exactly one event in exactly the sequence you expected.

Observation feeds a cache and a queue

A worker reads cached state and compares desired state with dependents; it may need another pass after cache lag.

Queue viewStep 1 of 3
Build a baselineThe informer lists and watches resources, storing observed objects in its local cache.. API list/watch, Current objects + changes, idle. Informer cache, Local eventual snapshot, active. Event handler, Enqueue namespace/name, idle. Key queue, Pending resource keys, idle. Reconcile worker, Reads current desired state, idleAPI list/watch: Current objects + changes
API list/watchCurrent objects + changesContext
Informer cache: Local eventual snapshot
Informer cacheLocal eventual snapshotIn focus
Event handler: Enqueue namespace/name
Event handlerEnqueue namespace/nameContext
Key queue: Pending resource keys
Key queuePending resource keysContext
Reconcile worker: Reads current desired state
Reconcile workerReads current desired stateContext
API list/watchCurrent objects + changesContext
Informer cacheLocal eventual snapshotIn focus
Event handlerEnqueue namespace/nameContext
Key queuePending resource keysContext
Reconcile workerReads current desired stateContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Read after write is stale
Build a baseline

The informer lists and watches resources, storing observed objects in its local cache.

Read the complete visual sequence
  1. Build a baseline

    The informer lists and watches resources, storing observed objects in its local cache.

  2. Keep handlers short

    A handler enqueues the object key instead of performing a slow external request in the watch path.

  3. Reconcile the latest observation

    A worker reads cached state and compares desired state with dependents; it may need another pass after cache lag.

Read after write is stale

A successful API write may not yet be reflected in the local cache.

  1. Write completed

    The worker created a dependent through the API, but its informer has not observed that change yet.

  2. Old observation returns

    The cache still reports no child. A blind second create can meet AlreadyExists.

  3. Recover by comparison

    Treat the conflict as new information, requeue appropriately, and reuse the stable child identity.

About this model

A single informer is shown; production controllers often depend on several caches and must synchronize them.

Walkthrough: one key may represent many changes

A workqueue can deduplicate pending keys and coordinate processing of a key so workers do not repeatedly execute the same pending item concurrently. If another change arrives while a key is being processed, the queue can arrange another pass after the current one finishes. This reduces unnecessary work but means an event stream is not a transaction log. Reconcile the current object’s desired replica count, rather than attempting to replay every historical increase and decrease.

Rate-limited queues let failures re-enter with delay. On success, a typical client-go worker forgets accumulated failure history; it always marks the item done after processing. On transient errors it adds the key with rate limiting. On permanent invalid input it records a useful condition and waits for a relevant change rather than retrying rapidly forever. Per-item backoff and overall throughput limits reduce pressure, but neither grants capacity to an overloaded downstream API.

Dedupe, delay, and retry safely

Success resets failure history; transient failure requeues with backoff instead of a tight loop.

Queue viewStep 1 of 3
Collapse redundant workMultiple pending notifications for one key can produce one current-state reconciliation.. Object changes, Many notifications per key, idle. Rate-limited queue, Dedupe pending key, active. Worker, One reconcile attempt, idle. Dependency API, May succeed before timeout, idle. Observed result, Conditions or successful state, idleObject changes: Many notifications per key
Object changesMany notifications per keyContext
Rate-limited queue: Dedupe pending key
Rate-limited queueDedupe pending keyIn focus
Worker: One reconcile attempt
WorkerOne reconcile attemptContext
Dependency API: May succeed before timeout
Dependency APIMay succeed before timeoutContext
Observed result: Conditions or successful state
Observed resultConditions or successful stateContext
Object changesMany notifications per keyContext
Rate-limited queueDedupe pending keyIn focus
WorkerOne reconcile attemptContext
Dependency APIMay succeed before timeoutContext
Observed resultConditions or successful stateContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Timeout after partial success
Collapse redundant work

Multiple pending notifications for one key can produce one current-state reconciliation.

Read the complete visual sequence
  1. Collapse redundant work

    Multiple pending notifications for one key can produce one current-state reconciliation.

  2. Attempt bounded work

    The worker reads current intent and issues repeatable operations with stable identities.

  3. Forget or rate-limit

    Success resets failure history; transient failure requeues with backoff instead of a tight loop.

Timeout after partial success

The next pass must discover work already committed by the dependency.

  1. Remote create committed

    The dependency created the resource but its response was lost.

  2. Retry is delayed

    Backoff protects the dependency while the key remains eligible for another attempt.

  3. Discover before replacing

    A stable identity lets reconciliation find the existing resource and complete status without duplication.

About this model

The scene illustrates standard workqueue behavior without prescribing one library version or retry duration.

The following configuration is used by the local mechanism exercise. The ConfigMap illustrates stable desired input, while the Lease is inspected separately in the leadership lesson. Reapplying a declaration is not a production informer implementation; it demonstrates why repeating an intention should preserve object identity.

apiVersion: v1
kind: ConfigMap
metadata:
  name: m15-repeat
  namespace: kv-lab
  labels:
    lab: m15
data:
  desired: stable
kubectl --context kind-kv-lab -n kv-lab apply -f repeat.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat -o jsonpath='{.metadata.uid}{"\n"}'
# Apply again; the object UID should remain the same.
kubectl --context kind-kv-lab -n kv-lab apply -f repeat.yaml
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat -o jsonpath='{.metadata.uid}{"\n"}'
kubectl --context kind-kv-lab -n kv-lab get configmap m15-repeat --watch --request-timeout=5s

Practice: reason about the crash boundary

Run the m15 local exercise, then connect it to the polling operator in m16. Draw a reconcile attempt that creates a ConfigMap, creates a Deployment, and updates status. Place a process crash after each write. The next attempt must discover and reuse an already-created object, finish missing work, and avoid inventing a new child name on every attempt. This is idempotence across partial progress, not merely making a function return the same string twice.

Inspect the example operator’s deliberately limited polling loop. Polling is useful for exposing the state comparison, but it does not provide shared informer efficiency, queue fairness or rate-limited retry machinery. A production conversion would use maintained client libraries, wait for required caches to synchronize, handle deleted objects, and measure queue depth, queue wait time, reconcile duration and error categories. Choose observations that distinguish a slow worker from an unhealthy API dependency.

Pitfalls: retries are a load generator

Do not retry every exception with no delay. A rejected schema, missing authorization, exhausted quota and temporary timeout require different remedies. If an external request timed out after the external system committed it, retrying a blind create can duplicate resources. Use a stable external identity, an idempotency key where available, and discovery before replacement. Cap concurrency so a short API outage does not produce an uncontrolled recovery surge. Logging every retry at maximum detail can itself consume storage and hide the first useful failure.

Check your understanding

Ten updates enqueue the same object before a worker runs. Must reconciliation execute ten times?

No. A deduplicating key queue may collapse pending work. The worker should read the latest desired state and converge toward it. If every individual transition matters to a business workflow, that workflow needs a durable event or transaction design beyond a controller workqueue.

Next, trace a specific controller that consumes unscheduled Pods. Its queue feeds a scheduling framework with feasibility checks, scoring and a separate binding outcome.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.