Appearance

DaemonSets, Jobs, and CronJobs

Choose per-node presence or finite completion, then design retries and schedules for duplicate execution.

What you will understand

  • Explain DaemonSet coverage on eligible nodes
  • Configure bounded Job retries and cleanup
  • Design CronJob work for missed or duplicate schedules

Builds on: StatefulSets and stable identity

Mental model: a replica can mean a node agent or a unit of completed work

A Deployment maintains a desired number of interchangeable long-running Pods. A DaemonSet instead maintains a Pod on each eligible node, commonly for log collection, network agents or node monitoring. Adding an eligible worker creates another placement opportunity; removing a worker removes that instance. Node selectors, affinity and taints determine eligibility, so “every node” is shorthand for every node the DaemonSet is configured and allowed to use.

DaemonSet Pods are still Pods with requests, security settings and lifecycle behavior. They consume node capacity, can fail probes and can be updated. The controller can automatically add certain tolerations appropriate to daemon workloads, but that does not justify making every node agent privileged. A log collector that needs host files has a different access boundary from an application metrics exporter that only needs an API. State exactly what host access the agent requires before choosing mounts and privileges.

Coverage follows eligible nodes

Show a DaemonSet creating one agent per eligible node and reacting to a newly added worker.

Ownership viewStep 1 of 4
Compute eligible nodesThe controller applies node selection and placement rules; the control node is excluded in this example.. DaemonSet logs, Selects pool=workers, active. Worker A, Eligible node, ready. Worker B, Eligible node, idle. Control node, Does not match pool, idle. Agent A, Node-local log collection, idle. Agent B, Node-local log collection, idleDaemonSet logs: Selects pool=workers
DaemonSet logsSelects pool=workersIn focus
Worker A: Eligible node
Worker AEligible nodeReady
Worker B: Eligible node
Worker BEligible nodeContext
Control node: Does not match pool
Control nodeDoes not match poolContext
Agent A: Node-local log collection
Agent ANode-local log collectionContext
Agent B: Node-local log collection
Agent BNode-local log collectionContext
DaemonSet logsSelects pool=workersIn focus
Worker AEligible nodeReady
Worker BEligible nodeContext
Control nodeDoes not match poolContext
Agent ANode-local log collectionContext
Agent BNode-local log collectionContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
New node has an untolerated taint
Compute eligible nodes

The controller applies node selection and placement rules; the control node is excluded in this example.

Read the complete visual sequence
  1. Compute eligible nodes

    The controller applies node selection and placement rules; the control node is excluded in this example.

  2. Create node-local instances

    Agent A is assigned to Worker A. Its purpose is coverage of that node rather than satisfying a generic replica count.

  3. Add another worker

    Worker B joins with matching labels and acceptable taints, creating another node that needs coverage.

  4. Reconcile the new instance

    Agent B runs on Worker B and consumes that worker resources. Requests and permissions still matter for infrastructure agents.

New node has an untolerated taint

A matching node label is insufficient for placement.

  1. Worker joins the pool

    Worker B has the correct pool label but also an additional dedicated taint.

  2. Agent cannot place

    The DaemonSet Pod has no appropriate toleration for this taint and cannot provide coverage there.

  3. Audit coverage and permissions

    Inspect desired/current/ready counts and node events before adding only the narrowly required toleration.

About this model

One agent Pod per eligible worker in steady state; update surge and transient overlap are omitted.

Walkthrough: completion needs a different controller

A Job manages finite work until the requested completions are achieved or failure criteria end it. restartPolicy: Never makes failed containers visible as failed Pods, while the Job controller can create replacements within its retry policy. OnFailure allows container restart within a Pod for suitable workloads. Neither option gives exactly-once business execution. A process can write to an external system, lose connectivity before acknowledging completion, and then be retried.

apiVersion: batch/v1
kind: Job
metadata:
  name: report-once
spec:
  backoffLimit: 2              # Bound retry attempts after failures.
  activeDeadlineSeconds: 120   # Bound total running time.
  ttlSecondsAfterFinished: 600 # Clean up later; export needed evidence first.
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: report
          image: busybox:1.36
          command: ["sh", "-c", "date -u; echo report-complete"]

For parallel work, Job parallelism controls concurrent Pods and completions controls required successful completions under the chosen completion mode. Indexed Jobs give each completion an index, useful for partitioned datasets. Partition identity still needs idempotent writes or deduplication in the destination. Logging a completion message is not a durable business transaction record, and the controller cannot infer whether an external side effect happened before a crash.

A CronJob creates Jobs according to a schedule. Its jobTemplate describes the Job, while concurrencyPolicy controls overlap among Jobs from that CronJob. Allow permits overlap, Forbid skips new work when a previous Job is still active, and Replace replaces the active run. This policy does not coordinate unrelated CronJobs. startingDeadlineSeconds limits how late a missed run may start, and .spec.timeZone makes the intended timezone explicit. Suspension pauses future scheduling but does not automatically erase active work.

A scheduled run contains retryable attempts

Follow CronJob scheduling into a Job, a failed attempt and a successful retry with duplicate side-effect risk.

Queue viewStep 1 of 4
Trigger a JobThe CronJob evaluates its schedule and concurrency policy, then creates a Job for an eligible run.. CronJob, Schedule + concurrencyPolicy, active. Job run, Completion and retry budget, active. Attempt 1, Writes then exits unsuccessfully, idle. Attempt 2, Replacement attempt, idle. Result store, Requires idempotency key, idleCronJob: Schedule + concurrencyPolicy
CronJobSchedule + concurrencyPolicyIn focus
Job run: Completion and retry budget
Job runCompletion and retry budgetIn focus
Attempt 1: Writes then exits unsuccessfully
Attempt 1Writes then exits unsuccessfullyContext
Attempt 2: Replacement attempt
Attempt 2Replacement attemptContext
Result store: Requires idempotency key
Result storeRequires idempotency keyContext
CronJobSchedule + concurrencyPolicyIn focus
Job runCompletion and retry budgetIn focus
Attempt 1Writes then exits unsuccessfullyContext
Attempt 2Replacement attemptContext
Result storeRequires idempotency keyContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Forbid skips overlapping schedule times
Trigger a Job

The CronJob evaluates its schedule and concurrency policy, then creates a Job for an eligible run.

Read the complete visual sequence
  1. Trigger a Job

    The CronJob evaluates its schedule and concurrency policy, then creates a Job for an eligible run.

  2. Run the first attempt

    The Job starts a Pod. It may produce an external side effect before its final exit status is known.

  3. Retry a failed attempt

    A failure within backoffLimit can create another attempt. The controller does not know whether the previous external write was committed.

  4. Complete idempotently

    The second attempt uses the same business work key so the destination can deduplicate the result, then Job completion is recorded.

Forbid skips overlapping schedule times

The previous Job remains active when the next schedule arrives.

  1. Previous run is active

    The current Job is still processing its bounded work.

  2. New scheduled time arrives

    Forbid does not create overlapping work from this CronJob while that Job remains active.

  3. Business work needs durable tracking

    If every interval must be processed, track intervals or queue items durably; concurrencyPolicy alone does not guarantee one result per schedule.

About this model

One CronJob and finite work; no exactly-once business guarantee.

Practice: inspect ownership and durable results

kubectl apply -f report-once.yaml
kubectl get jobs,pods
kubectl logs job/report-once
kubectl describe job report-once
# Inspect node-local coverage where the cluster already has DaemonSets.
kubectl get daemonsets -A
kubectl get pods -A -o wide

Expect the Job’s succeeded count to reach one for this finite command. The Pod can show Completed rather than Running; that is healthy for a successful Job. For a failure exercise, create a separate Job whose command exits with status 1 and observe its bounded retries and eventual failure. Do not repeatedly reapply the same completed Job expecting another run; create a new named Job or use kubectl create job --from=cronjob/NAME RUN_NAME for an installed CronJob.

Use the workload lab to compare a finite Pod with a managed Job. For a scheduling extension, write down what should happen if a one-minute CronJob takes three minutes with Forbid: not every scheduled time will become a completed report. Decide whether the business requirement is “latest result,” “one result per interval,” or “process every queued item,” then design durable work tracking accordingly.

Pitfalls: schedules are approximate triggers

Controller downtime, clock assumptions and long work duration can produce missed or duplicate scheduling situations. CronJob documentation explicitly calls for idempotent Jobs. Use a stable work key such as a reporting interval and a destination transaction or deduplication record. A node agent also needs bounded resources: a crashing log collector should not consume the entire worker while attempting to report another application’s failure.

Check your understanding

A Job writes a payment record, crashes before exiting successfully, and is retried. Does backoffLimit prevent a duplicate payment?

No. Backoff limits retry count; it does not make the external side effect atomic with Kubernetes completion status. Use an idempotency key or transactional application protocol so repeating the unit of work has the intended single business effect. The controller manages execution attempts rather than business exactly-once semantics.

Next, terminate long-running workloads deliberately and distinguish an ordinary retry from a voluntary maintenance disruption.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.