DaemonSets, Jobs, and CronJobs
Choose per-node presence or finite completion, then design retries and schedules for duplicate execution.
What you will understand
- Explain DaemonSet coverage on eligible nodes
- Configure bounded Job retries and cleanup
- Design CronJob work for missed or duplicate schedules
Builds on: StatefulSets and stable identity
Mental model: a replica can mean a node agent or a unit of completed work
A Deployment maintains a desired number of interchangeable long-running Pods. A DaemonSet instead maintains a Pod on each eligible node, commonly for log collection, network agents or node monitoring. Adding an eligible worker creates another placement opportunity; removing a worker removes that instance. Node selectors, affinity and taints determine eligibility, so “every node” is shorthand for every node the DaemonSet is configured and allowed to use.
DaemonSet Pods are still Pods with requests, security settings and lifecycle behavior. They consume node capacity, can fail probes and can be updated. The controller can automatically add certain tolerations appropriate to daemon workloads, but that does not justify making every node agent privileged. A log collector that needs host files has a different access boundary from an application metrics exporter that only needs an API. State exactly what host access the agent requires before choosing mounts and privileges.
Coverage follows eligible nodes
Show a DaemonSet creating one agent per eligible node and reacting to a newly added worker.
The controller applies node selection and placement rules; the control node is excluded in this example.
- DaemonSet logs to Worker Aeligible
- DaemonSet logs blocked toward Control nodeselector mismatch
Read the complete visual sequence
- Compute eligible nodes
The controller applies node selection and placement rules; the control node is excluded in this example.
- Create node-local instances
Agent A is assigned to Worker A. Its purpose is coverage of that node rather than satisfying a generic replica count.
- Add another worker
Worker B joins with matching labels and acceptable taints, creating another node that needs coverage.
- Reconcile the new instance
Agent B runs on Worker B and consumes that worker resources. Requests and permissions still matter for infrastructure agents.
New node has an untolerated taint
A matching node label is insufficient for placement.
- Worker joins the pool
Worker B has the correct pool label but also an additional dedicated taint.
- Agent cannot place
The DaemonSet Pod has no appropriate toleration for this taint and cannot provide coverage there.
- Audit coverage and permissions
Inspect desired/current/ready counts and node events before adding only the narrowly required toleration.
About this model
One agent Pod per eligible worker in steady state; update surge and transient overlap are omitted.
Walkthrough: completion needs a different controller
A Job manages finite work until the requested completions are achieved or failure criteria end it. restartPolicy: Never makes failed containers visible as failed Pods, while the Job controller can create replacements within its retry policy. OnFailure allows container restart within a Pod for suitable workloads. Neither option gives exactly-once business execution. A process can write to an external system, lose connectivity before acknowledging completion, and then be retried.
apiVersion: batch/v1
kind: Job
metadata:
name: report-once
spec:
backoffLimit: 2 # Bound retry attempts after failures.
activeDeadlineSeconds: 120 # Bound total running time.
ttlSecondsAfterFinished: 600 # Clean up later; export needed evidence first.
template:
spec:
restartPolicy: Never
containers:
- name: report
image: busybox:1.36
command: ["sh", "-c", "date -u; echo report-complete"]
For parallel work, Job parallelism controls concurrent Pods and completions controls required successful completions under the chosen completion mode. Indexed Jobs give each completion an index, useful for partitioned datasets. Partition identity still needs idempotent writes or deduplication in the destination. Logging a completion message is not a durable business transaction record, and the controller cannot infer whether an external side effect happened before a crash.
A CronJob creates Jobs according to a schedule. Its jobTemplate describes the Job, while concurrencyPolicy controls overlap among Jobs from that CronJob. Allow permits overlap, Forbid skips new work when a previous Job is still active, and Replace replaces the active run. This policy does not coordinate unrelated CronJobs. startingDeadlineSeconds limits how late a missed run may start, and .spec.timeZone makes the intended timezone explicit. Suspension pauses future scheduling but does not automatically erase active work.
A scheduled run contains retryable attempts
Follow CronJob scheduling into a Job, a failed attempt and a successful retry with duplicate side-effect risk.
The CronJob evaluates its schedule and concurrency policy, then creates a Job for an eligible run.
- CronJob to Job runscheduled run
Read the complete visual sequence
- Trigger a Job
The CronJob evaluates its schedule and concurrency policy, then creates a Job for an eligible run.
- Run the first attempt
The Job starts a Pod. It may produce an external side effect before its final exit status is known.
- Retry a failed attempt
A failure within backoffLimit can create another attempt. The controller does not know whether the previous external write was committed.
- Complete idempotently
The second attempt uses the same business work key so the destination can deduplicate the result, then Job completion is recorded.
Forbid skips overlapping schedule times
The previous Job remains active when the next schedule arrives.
- Previous run is active
The current Job is still processing its bounded work.
- New scheduled time arrives
Forbid does not create overlapping work from this CronJob while that Job remains active.
- Business work needs durable tracking
If every interval must be processed, track intervals or queue items durably; concurrencyPolicy alone does not guarantee one result per schedule.
About this model
One CronJob and finite work; no exactly-once business guarantee.
Practice: inspect ownership and durable results
kubectl apply -f report-once.yaml
kubectl get jobs,pods
kubectl logs job/report-once
kubectl describe job report-once
# Inspect node-local coverage where the cluster already has DaemonSets.
kubectl get daemonsets -A
kubectl get pods -A -o wide
Expect the Job’s succeeded count to reach one for this finite command. The Pod can show Completed rather than Running; that is healthy for a successful Job. For a failure exercise, create a separate Job whose command exits with status 1 and observe its bounded retries and eventual failure. Do not repeatedly reapply the same completed Job expecting another run; create a new named Job or use kubectl create job --from=cronjob/NAME RUN_NAME for an installed CronJob.
Use the workload lab to compare a finite Pod with a managed Job. For a scheduling extension, write down what should happen if a one-minute CronJob takes three minutes with Forbid: not every scheduled time will become a completed report. Decide whether the business requirement is “latest result,” “one result per interval,” or “process every queued item,” then design durable work tracking accordingly.
Pitfalls: schedules are approximate triggers
Controller downtime, clock assumptions and long work duration can produce missed or duplicate scheduling situations. CronJob documentation explicitly calls for idempotent Jobs. Use a stable work key such as a reporting interval and a destination transaction or deduplication record. A node agent also needs bounded resources: a crashing log collector should not consume the entire worker while attempting to report another application’s failure.
Check your understanding
A Job writes a payment record, crashes before exiting successfully, and is retried. Does backoffLimit prevent a duplicate payment?
No. Backoff limits retry count; it does not make the external side effect atomic with Kubernetes completion status. Use an idempotency key or transactional application protocol so repeating the unit of work has the intended single business effect. The controller manages execution attempts rather than business exactly-once semantics.
Next, terminate long-running workloads deliberately and distinguish an ordinary retry from a voluntary maintenance disruption.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise