Termination, disruption, and drain
Connect graceful shutdown to readiness, voluntary eviction budgets and node maintenance.
What you will understand
- Trace the Pod termination grace period and application signal handling
- Calculate whether a PDB permits a voluntary eviction
- Drain a node while recognizing failures that a PDB cannot prevent
Builds on: DaemonSets, Jobs, and CronJobs
Mental model: shutdown is a coordinated transition with a deadline
Deleting a Pod starts a termination workflow rather than instantly erasing every process and connection. The kubelet observes the grace period and arranges shutdown of containers. If configured, a preStop hook runs as part of that grace-period budget before the normal stop signal. The application should handle its stop signal, stop admitting new work, complete or hand off existing requests, and exit within the available time. If it does not exit in time, the runtime can force termination.
Endpoint changes occur alongside shutdown, not as one atomic global switch. A terminating endpoint is marked accordingly and ordinarily not ready for new normal traffic; implementations may use additional serving information for graceful handling. Load balancers, proxies and clients learn these changes asynchronously, and existing connections may remain active. A fixed sleep in preStop can allow propagation time, but it is not a substitute for a correct application shutdown protocol or a guarantee about every network device.
Traffic removal and process shutdown share a deadline
Trace deletion, endpoint readiness, preStop, stop signal and final process exit.
The API records deletion and a grace period. Endpoint state and node shutdown react asynchronously to that transition.
- API Pod to EndpointSlicetermination state
- API Pod to kubeletdeletion observed
Read the complete visual sequence
- Deletion begins
The API records deletion and a grace period. Endpoint state and node shutdown react asynchronously to that transition.
- Stop new traffic and run hook
Endpoint updates discourage new ordinary traffic, while preStop uses part of the same grace-period budget.
- Signal the process
After the hook, the runtime sends the normal stop signal. The application should finish or hand off in-flight work and stop accepting new work.
- Exit before the deadline
A graceful process exit completes shutdown. Existing connections and upstream propagation must be considered in the application design.
Hook consumes the grace budget
A long hook leaves insufficient time for application shutdown.
- Slow preStop starts
The grace countdown is already running while the hook waits.
- Application receives little time
The hook consumes most of the budget before the process can finish active work.
- Deadline forces termination
If shutdown does not complete within the available grace handling, the runtime can kill remaining processes. Increase budgets only with a measured shutdown protocol.
About this model
One ordinary container and a normal deletion; precise propagation time depends on components.
Walkthrough: allow maintenance without removing too many healthy replicas
A PodDisruptionBudget, or PDB, expresses the availability that should remain during covered voluntary disruptions. It is normally enforced through the Eviction API used by maintenance tooling. This PDB requires two healthy matching Pods to remain available.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-availability
spec:
minAvailable: 2
selector:
matchLabels:
app: api
With three healthy selected replicas, one voluntary eviction can usually proceed while respecting this requirement. Once only two healthy replicas remain, another covered eviction must wait until health recovers. The budget does not create replicas or enough capacity for replacements. A Deployment must maintain the intended count, and the scheduler needs somewhere eligible to put the replacements. A single-replica application with minAvailable: 1 can block ordinary maintenance indefinitely unless another healthy instance can be introduced.
A PDB does not prevent involuntary node crashes, kernel failures, storage failures or every direct Pod deletion. It is not a cap on unavailability caused by a Deployment’s own rolling-update strategy. Rollout controls such as maxUnavailable and maxSurge govern that controller’s behavior separately. Empty selectors in policy/v1 can select every Pod in the namespace, so keep the intended label boundary explicit.
Eviction asks permission from the current health budget
Follow cordon, PDB health evaluation, eviction and replacement during maintenance.
The node becomes unschedulable for ordinary new Pods. Existing workloads do not move merely because it is cordoned.
- Drain to Worker Aunschedulable
Read the complete visual sequence
- Cordon the worker
The node becomes unschedulable for ordinary new Pods. Existing workloads do not move merely because it is cordoned.
- Ask for an eviction
Drain requests voluntary eviction. With three healthy replicas and minimum two, the PDB can permit one disruption.
- Replace the removed replica
The controller sees fewer current replicas than desired and creates a replacement, which still needs suitable scheduling and storage.
- Wait for health before more removal
The replacement must become healthy before the next covered disruption can use a renewed budget.
A replica was already unready
Only two selected replicas are healthy before drain begins.
- Health budget exhausted
minAvailable=2 and healthy=2 leave no room for another covered healthy eviction.
- Eviction waits
The PDB rejects the voluntary removal that would reduce healthy availability below the budget.
- Repair health or plan capacity
Recover the unhealthy replica or add a viable replacement. A PDB cannot prevent node crashes or create capacity by itself.
About this model
Three selected replicas with minAvailable=2; ordinary Eviction API path and a controller-managed workload.
Practice: read the budget before touching the node
kubectl get pdb api-availability
kubectl describe pdb api-availability
kubectl get pods -l app=api -o wide
# Only use a disposable worker obtained from kubectl get nodes.
kubectl cordon LAB_WORKER
kubectl drain LAB_WORKER --ignore-daemonsets
# After maintenance and health checks:
kubectl uncordon LAB_WORKER
Cordon marks a node unschedulable for ordinary new placement. It does not move existing Pods. Drain attempts to remove appropriate workloads, normally using evictions that respect PDBs, while handling node-local and unmanaged workloads according to its flags. DaemonSet Pods are normally handled separately; --ignore-daemonsets permits drain to proceed without deleting those agents. Local emptyDir data and Pods without suitable controllers require conscious choices; do not add force or deletion flags merely to make an error disappear.
Watch the replacement Pods, events and PDB disruptionsAllowed while a disposable node is drained in the workload lab. If drain waits, first ask whether enough matching Pods are healthy, whether a replacement is Pending, and whether storage topology or affinity prevents relocation. A budget reporting zero permitted disruptions may be doing exactly what was requested. Increasing a timeout does not create missing capacity.
Pitfalls: a force action changes the safety boundary
Direct deletion, forceful termination and bypassing eviction checks can trade a blocked maintenance operation for an availability or data-integrity problem. This is particularly serious for stateful members on unreachable nodes: API deletion is not evidence that the old process has stopped writing. Establish node fencing or application-specific ownership before allowing a replacement writer. Test shutdown with long-lived requests so you can observe both application completion and traffic removal rather than relying only on a Pod phase.
Check your understanding
Three replicas have minAvailable=2, but one is already unready. Can drain evict another healthy replica immediately?
Normally no. Only two are currently healthy, so evicting another healthy member would violate the requested availability. Investigate and restore health or deliberately change the availability and maintenance plan. The PDB cannot compensate for the pre-existing unhealthy replica, and bypassing it removes the protection it was designed to provide.
Next, define who can perform these operations. Availability policy is meaningful only when identities and permissions also constrain who can change it or bypass its workflow.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise