Appearance

Debug the failing workload

Turn a visible Pod status into evidence about scheduling, image retrieval, process failure or runtime inspection.

What you will understand

  • Distinguish Pod phase from container waiting and termination reasons
  • Choose evidence for Pending, ImagePullBackOff and CrashLoopBackOff
  • Use ephemeral containers within explicit permission and runtime limits

Builds on: Protecting Secrets and tenant boundaries

Mental model: the status column is a clue, not a root cause

Start a workload investigation by locating its current lifecycle boundary. Has the API accepted the object? Has a controller created a Pod? Has the scheduler assigned a node? Has the runtime retrieved images and prepared volumes? Has the process started and failed? Each boundary has different evidence. Reading application logs cannot explain a scheduler that has never placed the Pod, and adding memory cannot repair an image name that does not exist.

Pending is a Pod phase, not a synonym for insufficient CPU. It can include time waiting for scheduling and time preparing containers after assignment. A CrashLoopBackOff display is a container waiting reason indicating repeated restarts with restart backoff; it is not a separate Pod phase. A process that repeatedly exits successfully can also enter this loop under restartPolicy: Always. ImagePullBackOff likewise describes delayed retry after image retrieval failure. Inspect the structured container states, Pod conditions and events so that a convenient CLI status string does not become the entire diagnosis.

Find the earliest boundary that has not succeeded

Distinguish an unscheduled Pod, image retrieval backoff and a repeatedly crashing process.

Lifecycle viewStep 1 of 4
Pending before assignmentAn unassigned Pod with FailedScheduling needs scheduler evidence, not application logs from a process that never ran.. Pod object, Accepted desired workload, warning. Scheduler, Node fit and assignment, warning. Image retrieval, Worker credentials + registry path, idle. App process, Startup and runtime behavior, idle. Evidence, Events, states and previous logs, activePod object: Accepted desired workload
Pod objectAccepted desired workloadWarning
Scheduler: Node fit and assignment
SchedulerNode fit and assignmentWarning
Image retrieval: Worker credentials + registry path
Image retrievalWorker credentials + registry pathContext
App process: Startup and runtime behavior
App processStartup and runtime behaviorContext
Evidence: Events, states and previous logs
EvidenceEvents, states and previous logsIn focus
Pod objectAccepted desired workloadWarning
SchedulerNode fit and assignmentWarning
Image retrievalWorker credentials + registry pathContext
App processStartup and runtime behaviorContext
EvidenceEvents, states and previous logsIn focus
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Liveness causes the restart loop
Pending before assignment

An unassigned Pod with FailedScheduling needs scheduler evidence, not application logs from a process that never ran.

Read the complete visual sequence
  1. Pending before assignment

    An unassigned Pod with FailedScheduling needs scheduler evidence, not application logs from a process that never ran.

  2. Assigned but image unavailable

    After node assignment, a pull error can produce ImagePullBackOff. Read the exact registry or network failure in events.

  3. Process exits repeatedly

    A retrieved image can start and then exit. Restart backoff produces CrashLoopBackOff as a container waiting reason.

  4. Choose evidence by boundary

    Previous logs explain a terminated process when available, while events explain placement and setup. Change only a demonstrated cause.

Liveness causes the restart loop

The application is slow to start and a probe repeatedly terminates it.

  1. Process starts

    The image and configuration are usable enough for the process to begin startup.

  2. Probe kills before readiness

    An overly aggressive liveness probe repeatedly triggers restarts during legitimate startup.

  3. Correct probe semantics

    Use a suitable startup probe and evidence-based timings; increasing memory would not repair this demonstrated probe failure.

About this model

One application container; displayed waiting reasons are not separate Pod phases.

Walkthrough: capture the evidence before changing the workload

kubectl get deployment,replicaset,pod -n learning -o wide
kubectl describe pod FAILING_POD -n learning
kubectl get events -n learning --sort-by=.metadata.creationTimestamp
kubectl logs FAILING_POD -n learning -c app --tail=100
kubectl logs FAILING_POD -n learning -c app --previous --tail=100
kubectl get pod FAILING_POD -n learning -o yaml

Replace the resource and container names with those shown by the first commands. --previous retrieves logs from the previous terminated container instance when available, which is useful when the current instance has not yet emitted the failing startup message. An error saying no previous instance exists is itself a limit of that evidence source. Capture logs promptly because log rotation, Pod replacement and node loss can remove local history. Avoid copying tokens, connection strings or sensitive user payloads into an incident note.

For an unscheduled Pending Pod, read FailedScheduling events and compare requests, labels, affinity, taints and volume constraints. If a node is already assigned, inspect image, volume and sandbox events instead. For ImagePullBackOff, distinguish image-not-found, authentication, authorization, rate limiting, network and certificate failures from the actual event message. The kubelet pulls images using its configured credentials and referenced pull Secrets; a successful docker pull on your laptop does not prove the worker has the same network path or identity.

For CrashLoopBackOff, read the last termination reason, exit code, restart count and previous logs. A failing liveness probe can repeatedly kill a process that would otherwise remain alive; a startup probe can protect a deliberately slow startup. An OOM kill has a different evidence pattern from an application configuration exception. Fix the smallest demonstrated cause and preserve the manifest revision so you can explain what changed.

Bring tools to the Pod network namespace

Follow authorization and admission before adding an ephemeral container for scoped investigation.

Flow viewStep 1 of 4
Ask a specific questionThe application has no shell, but the operator needs to test the listener from its network namespace.. Operator, Concrete diagnostic question, active. API subresource, pods/ephemeralcontainers permission, active. App Pod, Minimal image; no shell, idle. Debug container, Approved tools image, idle. Pod network, Shared localhost and routes, idleOperator: Concrete diagnostic question
OperatorConcrete diagnostic questionIn focus
API subresource: pods/ephemeralcontainers permission
API subresourcepods/ephemeralcontainers permissionIn focus
App Pod: Minimal image; no shell
App PodMinimal image; no shellContext
Debug container: Approved tools image
Debug containerApproved tools imageContext
Pod network: Shared localhost and routes
Pod networkShared localhost and routesContext
OperatorConcrete diagnostic questionIn focus
API subresourcepods/ephemeralcontainers permissionIn focus
App PodMinimal image; no shellContext
Debug containerApproved tools imageContext
Pod networkShared localhost and routesContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Debug permission is denied
Ask a specific question

The application has no shell, but the operator needs to test the listener from its network namespace.

Read the complete visual sequence
  1. Ask a specific question

    The application has no shell, but the operator needs to test the listener from its network namespace.

  2. Check permission and policy

    The subresource request must pass authorization and admission. Debugging does not automatically grant host access.

  3. Start tools beside the app

    The ephemeral container joins the Pod network and may target process visibility when supported by the runtime.

  4. Observe without replacing the app

    Test the relevant localhost port or namespace evidence. The debug container does not replace the application or become its deployment revision.

Debug permission is denied

The operator is not allowed to update the ephemeral-container subresource.

  1. Application remains unchanged

    The application Pod is still running or failing in its existing state.

  2. API rejects debug addition

    Authorization denies the subresource update before any new container can run.

  3. Use authorized evidence

    Read permitted logs and events or use an approved separate diagnostic Pod; request narrowly scoped access only if the task needs this capability.

About this model

Runtime supports the requested target visibility; privileges and process namespace access are not universal.

Practice: inspect a minimal image without changing its production contents

Ephemeral containers add a debug container to an existing Pod through a dedicated subresource. They are useful when the application image lacks a shell or diagnostic tools. They need permission and appropriate admission settings. They share the Pod’s network namespace, while process visibility targeting depends on the runtime and configuration. Seeing localhost does not automatically mean seeing every target process or its full filesystem.

# Use a throwaway Pod and an approved debug image in the local lab.
kubectl debug -it FAILING_POD -n learning \
  --image=busybox:1.36 --target=app
# From the debug shell, inspect connectivity without printing credentials.
# An exit returns to your local shell; it does not delete the application Pod.

A debug container is added to the Pod’s specification and is not a general replacement for the application container or a routine deployment mechanism. It cannot make a not-yet-running image supply logs, and the chosen image may still lack the tool you need. A copied debug Pod or a disposable separate client can be safer when changing the live Pod would introduce unnecessary risk. Use only the privileges required for the question being tested.

Pitfalls: repeated restarts can hide the first useful error

Deleting all failed Pods before capturing events and previous logs erases the clearest evidence. A controller may instantly create fresh Pods with the same defect, making the system appear active without progress. Compare the failing Pod’s image, configuration references and owning ReplicaSet to the intended revision. Avoid an unrelated bundle of fixes because a green Pod afterward would not establish which hypothesis was correct.

Check your understanding

A Pod shows Pending, has a nodeName, and reports FailedMount. Should you first lower its CPU request?

No. Assignment has already occurred, so the immediate failing boundary is volume setup on the selected node. Read the mount event, claim and driver evidence. Lowering CPU may create another revision but does not address a demonstrated mount failure.

Use the diagnostics lab to record a symptom, a hypothesis and the single observation that can disprove it. Next, apply the same boundary-by-boundary method to networking and storage.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.