Troubleshooting Kubernetes: Pod Status, Events, Logs, and Probes

Kubernetes troubleshooting is not guesswork; it is a fixed escalation path through four commands, each answering one question. kubectl get answers what state things are in: it lists objects with their status columns, so you can spot a Pod that is Pending, a Deployment showing 2/3 ready replicas, or a node that is NotReady. kubectl describe answers why: it prints the full object detail and, crucially, ends with the Events section, where the scheduler, the kubelet, and controllers record what they tried and why it failed. kubectl logs answers what the application itself said: the stdout and stderr of a container, which is where crashes, stack traces, and configuration errors show up. kubectl exec answers what it looks like from inside: it runs a command in a running container so you can inspect files, environment variables, or connectivity.

The KCNA exam tests this mapping directly. A question describes a symptom and asks which command you would run first, so anchor the pairs now.

Everything in the rest of this lesson hangs off this table: the phases and reasons you will see in kubectl get output, and the events and log patterns that explain them.

Every Pod has a phase, a single high-level summary of where it is in its lifecycle. There are exactly five, and the exam expects you to know all of them.

Two clarifications keep you out of trap answers. First, Pending does not mean broken: a Pod is Pending for a moment on every normal start while the scheduler picks a node and the kubelet pulls images. It only signals a problem when it persists, most often because no node has enough free CPU or memory, which shows up as a FailedScheduling event. Second, the STATUS column of kubectl get pods is not the phase. That column shows a more specific reason when one exists: a Pod in phase Pending may display ContainerCreating or ImagePullBackOff, and a Pod in phase Running may display CrashLoopBackOff because its container keeps restarting. Succeeded and Failed apply to run-to-completion workloads; a long-running server that dies does not move to Failed, its container is simply restarted inside the still-Running Pod.

So read kubectl get pods output as symptom, phase as lifecycle stage, and kubectl describe as the explanation.

The reasons in the STATUS column split into two families that the exam loves to contrast: the container image cannot be obtained, or the container runs and then dies.

ErrImagePull means the kubelet just tried to pull the image and failed: the image name is misspelled, the tag does not exist, or the registry is private and the Pod has no valid image pull credentials. After repeated failures the kubelet backs off between retries and the status becomes ImagePullBackOff - same root cause, now with an increasing delay between attempts. ErrImageNeverPull is the odd sibling: the Pod's image pull policy is set to Never, the image is not already present on the node, and the kubelet is forbidden from fetching it. ContainerCreating is not an error at all, just the normal setup step; it only matters when a Pod is stuck there, which usually points at a volume that cannot be mounted or attached, visible as FailedMount events.

CrashLoopBackOff is the other family entirely. The image pulled fine and the container started, but it exited, was restarted by the kubelet, exited again, and Kubernetes is now waiting an exponentially increasing backoff before the next restart. The cause lives inside the container: a bad configuration value, a missing dependency, a failing database connection, or a command that finishes immediately in a Pod meant to run forever. OOMKilled is a specific terminal reason: the container exceeded its memory limit and the kernel killed it, classically with exit code 137; if it keeps happening, it becomes a crash loop too.

kubectl describe pod checkout-7d9f prints everything the cluster knows about the Pod: its node, labels, IP, each container's state and last state with exit codes, its restart count, mounted volumes, and conditions. For troubleshooting, the payoff is the Events section at the bottom. Events are timestamped records emitted by the components that acted on the object - the scheduler saying where and whether it could place the Pod, the kubelet reporting image pulls, container starts, probe failures, and restarts.

A handful of event reasons cover most exam scenarios. FailedScheduling means the scheduler found no suitable node, and the message says why, typically insufficient CPU or insufficient memory on every candidate node. Failed or BackOff pull events accompany image problems and include the registry error message. FailedMount means a volume could not be attached or mounted, which is the classic cause of a Pod stuck in ContainerCreating. Unhealthy means a liveness or readiness probe failed, with the probe's error in the message. Killing follows repeated liveness failures, as the kubelet restarts the container.

Events are not only visible per object. kubectl get events lists them for a namespace, and sorting by creation time reconstructs the order in which things went wrong - useful when several objects are involved. Two properties matter for the exam: events are namespaced, and they are short-lived, retained for about an hour by default, so they explain recent history, not last week's incident. When a question says a Pod is Pending, or stuck creating, or being restarted and asks where to find the reason, the answer is the Events section of kubectl describe.

kubectl logs checkout-7d9f streams the stdout and stderr of a Pod's container - which is precisely why the cloud native convention is that applications log to stdout instead of files. This is where application-level failures live: unhandled exceptions, refused database connections, missing environment variables, bad flags. Two flags carry most of the diagnostic weight. If the Pod has multiple containers, you must pick one with -c, as in kubectl logs checkout-7d9f -c payment-sidecar; without it, kubectl targets the single container or the default one. And for a crash-looping container the current instance may have produced nothing yet, so kubectl logs checkout-7d9f --previous shows the output of the previous, terminated instance - the one that actually crashed. That flag is the canonical answer to how do you see why a container in CrashLoopBackOff died. Add -f to follow logs live, and -l app=checkout to aggregate logs across Pods matching a label selector.

When logs are not enough, kubectl exec opens a live view: kubectl exec -it checkout-7d9f -- sh starts an interactive shell inside the running container (again, -c selects a container in a multi-container Pod). From there you can check whether a config file was mounted, print environment variables, or test whether the container can resolve and reach a dependency. Its limits define when it applies: exec requires a running container, so it cannot help with a Pod that never starts, and minimal images may lack a shell entirely.

Keep the division of labor straight: describe tells you what Kubernetes did to the container, logs tell you what the process did inside it, and exec lets you poke around while it runs.

The kubelet uses three kinds of health probes, and the exam's favorite question is what happens when each one fails. A liveness probe asks: is this container still functioning, or is it wedged? When a liveness probe fails repeatedly, the kubelet kills and restarts the container. A readiness probe asks: can this container serve traffic right now? When a readiness probe fails, the Pod is marked not ready and is removed from the endpoints of every Service that selects it - traffic stops flowing to it, but the container is not restarted. When the probe passes again, the Pod is added back. A startup probe exists for slow-starting applications: while it is running, the other two probes are suspended, so a legacy app that needs two minutes to boot is not killed by an impatient liveness probe; only if the startup probe exhausts its failure budget is the container restarted.

All three run the same mechanisms - an HTTP GET, a TCP connection attempt, an exec command graded by exit code, or a gRPC health check - so the difference is purely in the consequence. Probe failures surface as Unhealthy events in kubectl describe, and a rising restart count with liveness failures in the events is a hint that an overly aggressive liveness probe, not the application, may be the real problem. The distinction to burn in: liveness restarts, readiness gates traffic.

Sometimes every Pod shows Running and the application still is not reachable. The next suspect is the Service. A Service forwards traffic only to Pods that match its label selector and are ready. If the selector does not match the Pods' labels - a typo like app: chekout, or labels changed in a new Deployment revision - the Service has no endpoints, and connections to it go nowhere. The same empty-endpoints symptom appears when the Pods exist but their readiness probes are failing. The check is direct: kubectl describe service checkout shows the selector and the current endpoints, and kubectl get endpoints checkout lists the backing addresses; an empty list is the tell. A subtler variant is a port mismatch, where the Service's target port does not match the port the container actually listens on - endpoints exist, but connections are refused.

The other above-Pod failure domain is the node. kubectl get nodes shows each node's status; NotReady means the node's kubelet has stopped reporting healthy to the control plane - the machine may be down, the kubelet stopped, or its network severed. Pods on a NotReady node cannot be managed, may show phase Unknown, and after a timeout are scheduled for replacement elsewhere by their controllers. kubectl describe node shows the node's conditions, including the pressure signals: MemoryPressure, DiskPressure, and PIDPressure. Under resource pressure the kubelet may evict Pods to protect the node, and the scheduler avoids placing new ones there. With the metrics server installed, kubectl top nodes and kubectl top pods show live CPU and memory usage, which is how you confirm a pressure or capacity theory with numbers.

Put the sequence together on a realistic case. A team ships a new version of the checkout service. Minutes later, users report errors. You run kubectl get pods and see the new Pods: one shows ImagePullBackOff, two show Running but with 0/1 in the READY column, and one older Pod is still Running and ready.

Take the image problem first. kubectl describe pod on the ImagePullBackOff Pod shows pull events failing with an authentication error against the private registry - the new Pod spec dropped the image pull secret. That is a spec fix, not an application fix. Next, the running-but-not-ready Pods: describe shows repeating Unhealthy events from the readiness probe, and because readiness gates traffic, those Pods have been removed from the Service's endpoints - which you confirm with kubectl get endpoints checkout, showing only the one old Pod. The readiness failures explain the user-facing errors: nearly all capacity is out of rotation. Why is readiness failing? kubectl logs on one of the new Pods prints a stack trace: the app cannot connect to the payments database because a required environment variable is empty in the new release. No amount of restarting will fix that; the Deployment needs its configuration corrected or the rollout rolled back.

Notice what made this fast: get surfaced three distinct symptoms, describe attributed each to a component action, endpoints confirmed the traffic impact, and logs found the application cause. No step was clever; each command answered exactly its own question. That chain - and knowing which link a question is pointing at - is the whole troubleshooting competency KCNA measures.