Your first debugging workflow
Turn a broken request into evidence at the API, workload, listener, Service, and DNS boundaries.
What you will understand
- Choose logs and events according to the failure stage
- Trace a request without restarting away useful evidence
- Verify both a targeted repair and the original caller path
Builds on: DNS and service exposure
Mental model: locate the first failed boundary
“A request failed” is a symptom, not a diagnosis. Start by identifying the caller, exact destination and port, response or error, time window, namespace, and recent change. Then separate the API object path from the application packet path. A successful kubectl get proves access to the API, not that the web server works. A successful Pod port-forward tests a selected backend through a debugging tunnel, not necessarily the Service’s ordinary forwarding path.
Use the narrowest useful evidence first. Confirm the context and namespace so you do not investigate a different workload with the same name. Inspect the owning Deployment, its ReplicaSets, and current Pods. Check whether Pods exist, have node assignments, started containers, and became ready. Each transition has a responsible component and different failure evidence. Avoid deleting the failing Pod before reading its status and logs; replacement can discard the very observation needed to explain the incident.
Choose evidence for the startup stage
Separate unscheduled objects, image preparation failures, and executed-process failures.
A Pending Pod without a node needs scheduling evidence. A bound Pending Pod can instead be preparing images or mounts.
- get -o wide + describe to Pod assignment + conditionsread fields
- get -o wide + describe to Component event reasonsread concrete reasons
Read the complete visual sequence
- Check assignment and conditions
A Pending Pod without a node needs scheduling evidence. A bound Pending Pod can instead be preparing images or mounts.
- Distinguish before-execution failure
ImagePullBackOff means image preparation is retrying. No application process from that failed start exists to write useful application logs.
- Inspect a process that did execute
For repeated exits, read termination reason, exit code, current logs, and previous-container logs if retained. These explain a different boundary than scheduling.
Failure: delete before collecting evidence
Failure: delete before collecting evidence
- A failing Pod contains useful state
The current Pod has termination details and prior container output. The replacement may not preserve these observations.
- Deletion discards a useful target
A controller creates Pod B, but commands aimed at A can no longer retrieve its ordinary live-object state or logs.
- Collect first, then change deliberately
Capture necessary status and redacted logs, identify the likely cause, and make a targeted repair. Use centralized observability for durable incident history.
About this model
One current Pod is inspected; events and prior logs exist only while retained by the environment.
Walkthrough: inspect startup before transport
Use service-demo from the preceding lessons in its dedicated lab namespace. Replace ACTUAL_POD_NAME with a current Pod name from the first listing. Keep your verified context and namespace explicit when multiple clusters are configured.
kubectl get deployment service-demo
kubectl get pods -l app=service-demo -o wide
kubectl describe pod ACTUAL_POD_NAME
kubectl logs ACTUAL_POD_NAME -c web --tail=100 --timestamps
# Useful only if that container has a previous terminated instance:
kubectl logs ACTUAL_POD_NAME -c web --previous --tail=100
kubectl get events --sort-by=.metadata.creationTimestamp
Events describe observations by components, such as failed scheduling, image pulling, mount preparation, or probe failures. They are not an indefinite audit log and may be aggregated or expire. Container logs are the application’s stdout/stderr made available through the runtime and node log path. They are not every file in the container, and they are not durable centralized storage by themselves. A failed image pull has no running application process to emit its startup logs.
An unassigned Pending Pod points toward scheduling evidence. A bound Pod waiting for an image points toward reference, credentials, and registry access. A restarting container points toward termination reason, exit code, logs, probe results, or memory pressure. CrashLoopBackOff describes delayed retries after repeated failures; it does not identify the underlying application defect. Read the immediate cause before choosing a repair.
Follow a broken request one hop at a time
Once a backend runs, inspect the Service selector, ports, and EndpointSlices. Compare endpoint addresses with actual Pod IPs, readiness conditions with probe evidence, and target ports with the actual listener. Test from the same caller network when possible. A probe from your laptop may use a different DNS resolver, network route, firewall policy, or credential path from the failing Pod.
kubectl get service service-demo -o yaml
kubectl get pods -l app=service-demo --show-labels
kubectl get endpointslices -l kubernetes.io/service-name=service-demo -o yaml
# Use the existing debug client from the prior lesson.
kubectl exec discovery-client -- nslookup service-demo
kubectl exec discovery-client -- wget -qO- http://service-demo:8080/
If name lookup fails, inspect the caller’s resolver configuration and cluster DNS path. If lookup succeeds, retain that evidence and test address/port reachability. If a direct Pod-IP request works from the same client but the Service request fails, focus on frontend port mapping, endpoint records, and Service forwarding. If both fail, the listener, network reachability, policy, or application becomes more likely. No single test proves every possible path healthy.
Narrow a broken Service request
Compare DNS, frontend routing, endpoint selection, and backend listener evidence.
The client asks for debug-mismatch:8080. Record whether failure is DNS, connection, or an HTTP response before changing anything.
- Debug client Pod to Cluster DNS answerresolve name
- Debug client Pod to debug-mismatch:8080attempt request
Read the complete visual sequence
- Reproduce from the actual caller network
The client asks for debug-mismatch:8080. Record whether failure is DNS, connection, or an HTTP response before changing anything.
- Inspect endpoint and listener boundaries
The Service has no matching ready endpoint because its selector is wrong. A direct backend or port-forward can still show the application itself responds.
- Repair and verify the original path
Correct only the selector, observe ready endpoint records, then repeat the same client request. API acceptance alone is not the repair proof.
Failure: a tunnel result is overgeneralized
Failure: a tunnel result is overgeneralized
- A Pod port-forward succeeds
The operator forwards directly to one Pod and receives a response. This proves that selected backend works through the tunnel.
- The original Service path still fails
The client Pod uses DNS and Service forwarding, which the tunnel bypassed. Its selector mismatch remains unfixed.
- Validate each distinct path
Keep the tunnel success as backend evidence. Repair the Service boundary and repeat the original client request before declaring recovery.
About this model
A normal ClusterIP Service and one same-namespace debug client are used; no external gateway is involved.
Practice: repair one deliberate mismatch
This valid Service object intentionally selects nothing in the previous workload. Save it as debug-mismatch.yaml and apply it only in the disposable lab namespace:
apiVersion: v1
kind: Service
metadata:
name: debug-mismatch
spec:
selector:
app: deliberately-unmatched # The intended Pods use app: service-demo.
ports:
- port: 8080
targetPort: http
Predict that the Service can receive a ClusterIP and DNS record while lacking usable endpoints. Verify each prediction, change only the selector to app: service-demo, apply, and repeat the original client request to http://debug-mismatch:8080/. The repair should change the backend evidence and the request result. Merely seeing “configured” after apply is not the success condition.
Pitfalls
A minimal image may have no shell, nslookup, or curl. Missing debug tools do not establish an application failure. Use an approved client Pod or later an ephemeral-debug-container workflow rather than modifying the production image during an incident. Redact credentials and user data from logs before sharing them. Also avoid assuming every HTTP 500 comes from the Pod you intended; proxies and upstream dependencies can produce their own responses.
Check your understanding
The Pod answers over port-forward, but the Service request fails from a client Pod. Is the Service proven healthy?
No. Port-forward reaches a selected Pod through a different path. Inspect Service selection, endpoint readiness, client-facing and target ports, DNS, and node forwarding from the failing caller. Preserve the successful backend test as a useful narrowing result, then verify the repaired original path and several representative requests.
Complete the module lab by recording symptom, evidence, responsible layer, targeted change, and repeated verification. You are ready to study resources and placement with this same boundary-by-boundary method.
Primary documentation
Go to the source
Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise