Troubleshooting Kubernetes: Pod Status, Events, Logs, and Probes

Kubernetes troubleshooting is not guesswork; it is a fixed escalation path through four commands, each answering one question, plus a fifth for resource numbers. kubectl get answers what state things are in: it lists objects with their status columns, so you can spot a Pod that is Pending, a Deployment showing 2/3 ready replicas, or a node that is NotReady. kubectl describe answers why: it prints the full object detail, including the configured resource requests and limits, and ends with the Events section, where the scheduler, the kubelet, and controllers record what they tried and why it failed. kubectl logs answers what the application itself said: the stdout and stderr of a container, which is where crashes, stack traces, and configuration errors show up. kubectl exec answers what it looks like from inside: it runs a command in a running container so you can inspect files, environment variables, or connectivity. kubectl top answers how much is actually being used: it reports live CPU and memory consumption read from the Metrics API, which the metrics-server add-on supplies - without metrics-server, top returns an error rather than numbers.

The KCNA exam tests this mapping directly. A question describes a symptom and asks which command you would run first, so anchor the pairs now.

Everything in the rest of this lesson hangs off this table: the phases and reasons you will see in kubectl get output, and the events and log patterns that explain them.

Every Pod has a phase, a single high-level summary of where it is in its lifecycle. There are exactly five, and the exam expects you to know all of them.

Phases describe the Pod; each container inside it has its own, separate state, and there are only three: Waiting (not running yet - pulling an image, applying configuration, or sitting in a restart backoff), Running, and Terminated (finished or failed, with an exit code and a reason such as Completed, Error, or OOMKilled). So Waiting, Running, and Terminated are never Pod phases, and Pending or Succeeded are never container states.

Three clarifications keep you out of trap answers. First, Pending does not mean broken: a Pod is Pending for a moment on every normal start while the scheduler picks a node and the kubelet pulls images. It only signals a problem when it persists, most often because no node has enough free CPU or memory, which shows up as a FailedScheduling event. Second, the STATUS column of kubectl get pods is not the phase. That column shows a more specific reason when one exists, usually taken from a container state: a Pod in phase Pending may display ContainerCreating or ImagePullBackOff (ContainerCreating on its own is not an error - it is the normal setup step while images are pulled and volumes mounted, and only matters when a Pod stays stuck there), a Pod in phase Running may display CrashLoopBackOff because its container keeps restarting, and a Pod in phase Succeeded displays Completed - the reason its container terminated with. Third, Succeeded and Failed apply to run-to-completion workloads and to evicted Pods; a long-running server whose process dies does not move to Failed, its container is simply restarted inside the still-Running Pod.

Two node-level failures also separate cleanly by phase. A Pod on a node that has stopped reporting shows Unknown: nobody can say what it is doing. A Pod that the kubelet deliberately evicted to relieve node pressure ends in Failed with reason Evicted: the node is still reporting, and it reported that it removed the Pod. So read kubectl get pods output as symptom, phase as lifecycle stage, and kubectl describe as the explanation.

The reasons in the STATUS column split into two families that the exam loves to contrast: the container image cannot be obtained, or the container runs and then dies.

Image pulls happen on the node, by the kubelet, and only after the scheduler has bound the Pod to that node. A Pod still stuck with FailedScheduling has therefore not attempted any pull yet - an image problem cannot be its cause, and fixing a bad image name will not get it placed. Once bound, the kubelet follows the container's imagePullPolicy:

When the field is omitted, the default is Always for an image with the :latest tag or no tag, and IfNotPresent otherwise. ErrImagePull means the kubelet just tried to pull the image and failed: the image name is misspelled, the tag does not exist, the registry is unreachable, or it is private and no valid credentials were offered. After repeated failures the kubelet backs off between retries and the status becomes ImagePullBackOff - same root cause, now with an increasing delay between attempts. Registry credentials do not have to live in the Pod: besides the Pod's imagePullSecrets (or those attached to its ServiceAccount), a node can supply them itself through a kubelet credential provider plugin or the node's container runtime configuration, which is why the same Pod spec can pull on one node and fail on another.

CrashLoopBackOff is the other family entirely. The image pulled fine and the container started, but it exited, was restarted by the kubelet, exited again, and Kubernetes is now waiting an exponentially increasing backoff before the next restart. The cause lives inside the container: a bad configuration value, a missing dependency, a failing database connection, or a command that finishes immediately in a Pod meant to run forever. OOMKilled is a specific terminated reason: the container exceeded its own memory limit and the kernel killed it, classically with exit code 137; if it keeps happening, it becomes a crash loop too. A third family stops between the two: CreateContainerConfigError, where the image is present but the container cannot be created because its configuration points at something missing (covered in the next section).

Configuration and storage problems fail at different stages of a Pod's start, and the stage tells you which object is wrong. The scheduler places the Pod first, then the kubelet mounts its volumes, then it creates each container with its environment, then the container runs. A missing or unusable dependency surfaces at whichever stage first needs it.

The near-twins to separate are the two ways a missing ConfigMap or Secret can hurt. Referenced as environment variables, it blocks container creation with CreateContainerConfigError - the kubelet cannot build the container's environment. Mounted as a volume, it stalls the Pod in ContainerCreating with FailedMount events instead. In both cases the Pod waits and retries; creating the missing object (or marking the reference optional: true) lets it continue without editing the Pod. Neither is a crash loop: the application never ran, so kubectl logs has nothing to show, and the answer is in the Events section of kubectl describe.

Memory and CPU limits are enforced very differently, and that difference explains several symptoms. A memory limit is a hard ceiling enforced per container by the kernel: when a container's usage exceeds its own limit, it is killed with reason OOMKilled (exit code 137) and restarted according to the Pod's restart policy. This happens even when the node has plenty of free memory - the limit belongs to the container, not the node. A CPU limit never kills anything: a container that tries to use more CPU than its limit is throttled, simply getting less CPU time and running slower.

Throttling can still cause restarts indirectly. If a throttled application answers its health endpoint too slowly, its liveness probe times out; describe shows Unhealthy events whose message reports a timeout (for an HTTP probe, typically "context deadline exceeded"), followed by a Killing event as the kubelet restarts the container. The container's Last State then shows an ordinary termination reason such as Error - not OOMKilled - which is how you tell a probe-driven restart from a memory kill. Comparing kubectl top pod (live usage) with the limits shown by kubectl describe pod confirms which resource is at its ceiling.

Node-pressure eviction is a third, separate mechanism. When the node itself runs short, the kubelet evicts Pods to protect it. Its eviction signals are memory, disk (node and image filesystems), and process IDs - CPU is not an eviction signal, because CPU can always be shared out by throttling. An evicted Pod is not restarted in place: it ends in phase Failed with reason Evicted, and its controller (a Deployment's ReplicaSet, for example) creates a replacement that the scheduler places elsewhere.

kubectl describe pod checkout-7d9f prints everything the cluster knows about the Pod: its node, labels, IP, each container's state and last state with exit codes, its restart count, mounted volumes, and conditions. For troubleshooting, the payoff is the Events section at the bottom. Events are timestamped records emitted by the components that acted on the object - the scheduler saying where and whether it could place the Pod, the kubelet reporting image pulls, container starts, probe failures, and restarts.

A handful of event reasons cover most exam scenarios. FailedScheduling means the scheduler found no suitable node, and the message says why, typically insufficient CPU or insufficient memory on every candidate node. Failed or BackOff pull events accompany image problems and include the registry error message. FailedMount means a volume could not be attached or mounted, which is the classic cause of a Pod stuck in ContainerCreating. Unhealthy means a liveness or readiness probe failed, with the probe's error in the message. Killing follows repeated liveness failures, as the kubelet restarts the container.

Events are not only visible per object. kubectl get events lists them for a namespace, and sorting by creation time reconstructs the order in which things went wrong - useful when several objects are involved. Two properties matter for the exam: events are namespaced, and they are short-lived, retained for about an hour by default, so they explain recent history, not last week's incident. When a question says a Pod is Pending, or stuck creating, or being restarted and asks where to find the reason, the answer is the Events section of kubectl describe.

kubectl logs checkout-7d9f streams the stdout and stderr of a Pod's container - which is precisely why the cloud native convention is that applications log to stdout instead of files. This is where application-level failures live: unhandled exceptions, refused database connections, missing environment variables, bad flags. Three flags carry most of the diagnostic weight.

• Container choice. If the Pod has multiple containers, pick one with -c, as in kubectl logs checkout-7d9f -c payment-sidecar, or read every container at once with --all-containers. Without either, kubectl uses the Pod's only container or its default container.
• Previous instance. For a crash-looping container the current instance may have produced nothing yet, so kubectl logs checkout-7d9f --previous shows the output of the previous, terminated instance - the one that actually crashed. That flag is the canonical way to see why a container in CrashLoopBackOff died.
• Time and length filters. --since=1h and --tail=100 only trim the output of whichever instance you are reading. Without --previous that is the current instance, so widening the time window never reaches a run that crashed and was restarted.

Two more flags select differently and are easy to confuse: -l app=checkout selects Pods by label and aggregates logs across all of them, while -c selects a container inside one Pod. Add -f to follow logs live.

When logs are not enough, kubectl exec opens a live view: kubectl exec -it checkout-7d9f -- sh starts an interactive shell inside the running container (again, -c selects a container in a multi-container Pod). From there you can check whether a config file was mounted, print environment variables, or test whether the container can resolve and reach a dependency. Its limits define when it applies: exec requires a running container, so it cannot help with a Pod that never starts, and minimal images may lack a shell entirely.

Keep the division of labor straight: describe tells you what Kubernetes did to the container, logs tell you what the process did inside it, and exec lets you poke around while it runs.

The kubelet uses three kinds of health probes, and the exam's favorite question is what happens when each one fails. A liveness probe asks: is this container still functioning, or is it wedged? When a liveness probe fails repeatedly, the kubelet kills and restarts the container. A readiness probe asks: can this container serve traffic right now? When a readiness probe fails, the Pod is marked not ready and is removed from the endpoints of every Service that selects it - traffic stops flowing to it, but the container is not restarted. When the probe passes again, the Pod is added back. A startup probe exists for slow-starting applications: while it is running, the other two probes are suspended, so a legacy app that needs two minutes to boot is not killed by an impatient liveness probe; only if the startup probe exhausts its failure budget is the container restarted.

Two other start-time features look similar but do not hold off health checks. An init container runs to completion before the app container starts - useful for a setup step such as waiting for a database - but once the app container starts, its liveness probe starts on its normal schedule, however long the app itself takes to initialise. A postStart hook is a lifecycle hook, not a probe: it runs a handler right after the container is created, gives no readiness or liveness signal, and kills the container if it fails. Only the startup probe tells the kubelet to wait for the app inside the container.

All three probes run the same mechanisms - an HTTP GET, a TCP connection attempt, an exec command graded by exit code, or a gRPC health check - so the difference is purely in the consequence. Probe failures surface as Unhealthy events in kubectl describe, and a rising restart count with liveness failures in the events is a hint that an overly aggressive liveness probe, not the application, may be the real problem. The distinction to burn in: liveness restarts, readiness gates traffic.

Sometimes every Pod shows Running and the application still is not reachable. The next suspect is the Service. A Service forwards traffic only to Pods that match its label selector and are ready. If the selector does not match the Pods' labels - a typo like app: chekout, or labels changed in a new Deployment revision - the Service has no endpoints, and connections to it go nowhere. The same empty-endpoints symptom appears when the Pods exist but their readiness probes are failing. The check is direct: kubectl describe service checkout shows the selector and the current endpoints, and kubectl get endpoints checkout (or kubectl get endpointslices -l kubernetes.io/service-name=checkout, since the v1 Endpoints API is deprecated in favour of EndpointSlices from Kubernetes v1.33) lists the backing addresses; an empty list is the tell.

Ports cause two look-alike failures that the READY column separates. A container's containerPort is mostly informational: declaring it, or declaring the wrong number, does not open, close, or probe anything, and it has no effect on readiness or Service routing - unless something refers to it by name, such as a Service's targetPort or a probe's port. A wrong Service targetPort leaves the Pods READY 1/1 and listed as ready endpoints, yet connections through the Service are refused because traffic is sent to a port nothing listens on. A readiness probe pointed at the wrong port instead shows READY 0/1 with RESTARTS 0 (readiness never restarts anything) and leaves the Service with no ready endpoints.

The other above-Pod failure domain is the node. kubectl get nodes shows each node's status; NotReady means the node's kubelet has stopped reporting healthy to the control plane - the machine may be down, the kubelet stopped, or its network severed. Pods on a node that stops reporting cannot be managed, may show phase Unknown, and after a timeout are replaced elsewhere by their controllers. kubectl describe node shows the node's conditions, including the pressure signals: MemoryPressure, DiskPressure, and PIDPressure. Under resource pressure the kubelet evicts Pods to protect the node - those end in phase Failed with reason Evicted, not Unknown - and the scheduler avoids placing new ones there. With metrics-server installed, kubectl top nodes and kubectl top pods show live CPU and memory usage, which is how you confirm a pressure or capacity theory with numbers.

Put the sequence together on a realistic case. A team ships a new version of the checkout service. Minutes later, users report errors. You run kubectl get pods and see the new Pods: one shows ImagePullBackOff, two show Running but with 0/1 in the READY column, and one older Pod is still Running and ready.

Take the image problem first. kubectl describe pod on the ImagePullBackOff Pod shows pull events failing with an authentication error against the private registry - the new Pod spec dropped the image pull secret. That is a spec fix, not an application fix. Next, the running-but-not-ready Pods: describe shows repeating Unhealthy events from the readiness probe, and because readiness gates traffic, those Pods have been removed from the Service's endpoints - which you confirm with kubectl get endpoints checkout, showing only the one old Pod. The readiness failures explain the user-facing errors: nearly all capacity is out of rotation. Why is readiness failing? kubectl logs on one of the new Pods prints a stack trace: the app cannot connect to the payments database because a required environment variable is empty in the new release. No amount of restarting will fix that; the Deployment needs its configuration corrected or the rollout rolled back.

Notice what made this fast: get surfaced three distinct symptoms, describe attributed each to a component action, endpoints confirmed the traffic impact, and logs found the application cause. No step was clever; each command answered exactly its own question. That chain - and knowing which link a question is pointing at - is the whole troubleshooting competency KCNA measures.