Appearance

CPU throttling, OOM, and eviction

Identify the resource boundary before choosing a remedy.

What you will understand

  • Distinguish CPU quota throttling from memory OOM termination.
  • Explain node-pressure eviction and its evidence.
  • Relate requests, limits and QoS without promising immunity.

Builds on: Topology and resource governance, Informers, caches, and workqueues

Mental model: different limits fail differently

A CPU limit can cause throttling when the container exhausts its allowed execution quota. The process is delayed rather than automatically killed for crossing a CPU limit. Latency and request queues can grow even while average CPU graphs look modest, particularly when sampling hides short bursts. A CPU request instead informs scheduling and relative allocation under contention; it is not a promise that every request receives a dedicated physical core.

Memory has a different failure boundary. A container that cannot satisfy allocations within its enforced memory limit can be terminated by the kernel OOM mechanism. Kubernetes can report OOMKilled in container termination state, and restart policy determines what happens next. Memory requests guide placement and eviction considerations; they do not reserve every byte against every failure. A process can also fail allocations itself, so an application exception is not automatically evidence of a kernel OOM kill.

CPU delays work; memory can terminate it

The enforced memory boundary can trigger a kernel OOM kill recorded in container termination state.

Chart viewStep 1 of 3
Requests inform placementCPU and memory requests inform scheduling; they are distinct from enforced runtime limits.. CPU-limited process, Burst exceeds available quota, active. CPU quota, Execution delayed, idle. Memory-limited process, Allocation exceeds enforced limit, idle. Kernel OOM, May terminate container, idle. Container status, Reason + restart evidence, idleCPU-limited process: Burst exceeds available quota
CPU-limited processBurst exceeds available quotaIn focus
CPU quota: Execution delayed
CPU quotaExecution delayedContext
Memory-limited process: Allocation exceeds enforced limit
Memory-limited processAllocation exceeds enforced limitContext
Kernel OOM: May terminate container
Kernel OOMMay terminate containerContext
Container status: Reason + restart evidence
Container statusReason + restart evidenceContext
CPU-limited processBurst exceeds available quotaIn focus
CPU quotaExecution delayedContext
Memory-limited processAllocation exceeds enforced limitContext
Kernel OOMMay terminate containerContext
Container statusReason + restart evidenceContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Misdiagnose throttling as OOM
Requests inform placement

CPU and memory requests inform scheduling; they are distinct from enforced runtime limits.

Read the complete visual sequence
  1. Requests inform placement

    CPU and memory requests inform scheduling; they are distinct from enforced runtime limits.

  2. CPU quota is exhausted

    CPU work waits for execution budget, potentially increasing latency without terminating the process.

  3. Memory allocation cannot continue

    The enforced memory boundary can trigger a kernel OOM kill recorded in container termination state.

Misdiagnose throttling as OOM

A slow process needs evidence before a memory-limit change.

  1. Latency rises

    The CPU-limited container remains Running while requests queue.

  2. No memory termination

    There is no OOMKilled evidence; changing memory does not address CPU quota exhaustion.

  3. Inspect the right signal

    Correlate quota throttling with latency and concurrency before changing CPU configuration.

About this model

Linux cgroup-style CPU quota and memory behavior is summarized; kernel and runtime accounting affect exact observations.

Walkthrough: pressure can remove a whole Pod

Node-pressure eviction is the kubelet reclaiming resources when thresholds for memory, disk space, inodes or other supported signals are met. It differs from a container reaching its own memory limit. An evicted Pod is normally marked Failed with an eviction reason and message; a workload controller may create a replacement. The replacement may land on another node, but only if scheduling and capacity allow it. Eviction is not equivalent to a container restart inside the same Pod identity.

QoS classes help characterize resource configuration, but Guaranteed is not immunity from every eviction or host failure. Eviction ranking considers whether usage exceeds requests, Pod priority and relative excess for the pressured resource. Disk and inode pressure have their own accounting complications. Preserve the termination message, events, node conditions and container lastState before a cleanup or new rollout hides the evidence. A single exit code without context is usually insufficient to distinguish all these paths.

Node pressure differs from a container OOM

A controller can create a new Pod, but it still needs a feasible node and does not retain the evicted UID.

Lifecycle viewStep 1 of 3
Observe node pressureThe kubelet detects a configured resource pressure signal and attempts reclaim.. Worker node, Memory / disk / inode pressure, active. Kubelet, Reclaim and eviction decisions, idle. Selected Pod, May exceed resource requests, idle. Replacement Pod, New UID if controller creates it, idleWorker node: Memory / disk / inode pressure
Worker nodeMemory / disk / inode pressureIn focus
Kubelet: Reclaim and eviction decisions
KubeletReclaim and eviction decisionsContext
Selected Pod: May exceed resource requests
Selected PodMay exceed resource requestsContext
Replacement Pod: New UID if controller creates it
Replacement PodNew UID if controller creates itContext
Worker nodeMemory / disk / inode pressureIn focus
KubeletReclaim and eviction decisionsContext
Selected PodMay exceed resource requestsContext
Replacement PodNew UID if controller creates itContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Replacement also cannot fit
Observe node pressure

The kubelet detects a configured resource pressure signal and attempts reclaim.

Read the complete visual sequence
  1. Observe node pressure

    The kubelet detects a configured resource pressure signal and attempts reclaim.

  2. Choose eviction when needed

    If reclaim is insufficient, the kubelet ranks eligible Pods using relevant usage, requests and priority.

  3. Replacement needs capacity

    A controller can create a new Pod, but it still needs a feasible node and does not retain the evicted UID.

Replacement also cannot fit

Eviction does not create additional cluster capacity.

  1. Pressure removes a Pod

    A node-pressure eviction produces Pod failure evidence distinct from a container restart.

  2. Controller requests replacement

    Desired replicas remain, so a new Pod is created.

  3. Capacity remains exhausted

    The new Pod can remain Pending or encounter pressure again until demand or capacity changes.

About this model

Eviction thresholds and ranking depend on the pressured resource and kubelet configuration; QoS is not universal protection.

This complete memory-limited Pod is an optional small demonstration similar to the m17 download. It intentionally allocates memory through a memory-backed volume so the allocation is charged to the container. Its local outcome depends on runtime and kernel accounting; the downloadable script checks for OOMKilled rather than claiming every environment behaves identically.

apiVersion: v1
kind: Pod
metadata:
  name: m17-memory
  namespace: kv-lab
  labels:
    lab: m17
spec:
  restartPolicy: Never
  containers:
    - name: fill
      image: busybox:1.37.0
      command: [dd, if=/dev/zero, of=/memory/blob, bs=1M, count=96]
      resources:
        requests:
          memory: 16Mi
          cpu: 10m
        limits:
          memory: 32Mi
          cpu: 100m
      volumeMounts:
        - name: memory
          mountPath: /memory
  volumes:
    - name: memory
      emptyDir:
        medium: Memory
kubectl --context kind-kv-lab -n kv-lab get pod m17-oom -o yaml
kubectl --context kind-kv-lab -n kv-lab describe pod m17-oom
kubectl --context kind-kv-lab get nodes
kubectl --context kind-kv-lab -n kv-lab get events --sort-by=.metadata.creationTimestamp

Practice: explain the observed termination

Run the m17 local exercise rather than generating node-wide pressure on your laptop. The scripted Pod has a bounded memory limit and the script waits for OOMKilled. Inspect the exact container termination reason and compare it with the Pod phase. If the expected reason does not appear, inspect logs and runtime accounting; do not automatically increase allocation until the host becomes unstable. This is a container-limit exercise, not a proof of production eviction ordering.

For a throttling investigation, collect CPU quota/throttling metrics from the runtime or monitoring system and correlate them with application latency, concurrency and queue length. kubectl top requires a metrics provider and gives a sampled utilization view, not the complete throttling history. For eviction, inspect node pressure conditions and kubelet events before changing container memory limits. Choose the remedy at the boundary the evidence actually identifies.

Pitfalls: bigger limits can move the bottleneck

Increasing memory limits without increasing requests can let Pods consume more than the node can safely support. Increasing CPU limits may improve one container’s burst behavior while increasing contention for neighbors. A memory-backed emptyDir consumes memory and can contribute to memory pressure; a disk-backed emptyDir has different pressure behavior. Include startup peaks, caches, heap behavior and realistic load when setting requests. Avoid using restart count alone as the health signal: repeated replacement Pods can reset the visible history under new UIDs.

Check your understanding

A request becomes slow at its CPU limit but the container stays Running. Is that inconsistent?

No. CPU quota exhaustion commonly throttles execution rather than killing the container. Check throttling and queueing evidence. OOMKilled and node-pressure eviction are different mechanisms with different termination and event evidence.

Next, examine how an autoscaler interprets CPU usage relative to requests. Resource configuration changes both placement and the feedback signal used to scale.

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.