Debugging Application Delivery: Stuck Rollouts, Failed Releases, and Rollbacks

Generic Pod troubleshooting asks why a container is crashing. Delivery debugging asks a different question: you shipped a new version - did it arrive, and is it serving? The failure modes are specific to the rollout machinery: a new ReplicaSet that never becomes healthy, a rollout frozen halfway, a Helm upgrade stuck in a failed state, or a release that technically completed but doubled the error rate. The KCNA delivery domain probes this distinction, so keep the two mindsets separate.

The single most important fact is that a rolling update is gated on readiness. The Deployment controller creates a new ReplicaSet and scales it up in steps, but it only continues - and only scales the old ReplicaSet down - as new Pods pass their readiness probes and count as available. The bounds maxSurge and maxUnavailable guarantee a minimum serving capacity at every step.

That gating is what makes delivery failures survivable. If the new Pods never become Ready, the rollout simply stops making progress: the new ReplicaSet sits with a few broken Pods, and the old ReplicaSet stays up and keeps serving traffic. Users are usually unaffected by a completely broken release; what you have is a stuck rollout, not an outage. Your job as the debugger is to notice the hang, find out why the new Pods are not Ready, and either fix forward or roll back. The rest of this lesson is that workflow, in order.

The first tool is kubectl rollout status deployment/web. It blocks and prints progress while the rollout proceeds, exits successfully when the Deployment reaches its desired state, and simply keeps waiting - printing something like Waiting for deployment web rollout to finish: 1 out of 3 new replicas have been updated... - when the rollout is stuck. A status command that never returns is your first signal that the new version is not becoming healthy.

Next, look at the Deployment summary and its ReplicaSets:

kubectl get deployment web
kubectl get replicasets
kubectl describe deployment web

kubectl get deployment shows the columns READY, UP-TO-DATE, and AVAILABLE. During a healthy rollout UP-TO-DATE climbs to the desired count and AVAILABLE follows; in a stuck rollout UP-TO-DATE stalls at a low number while AVAILABLE reflects the old Pods still serving. kubectl get replicasets makes the mechanism visible: mid-rollout you will see two ReplicaSets for the Deployment - the old one still holding most replicas and the new one with a small number that refuse to become ready. That two-ReplicaSet picture is the signature of a stalled rolling update and a favorite exam scenario.

kubectl describe deployment adds the events history - each scale-up and scale-down decision the controller made - and the conditions block covered in the next section. From there the trail leads downward: the Deployment tells you the rollout is stuck, but the reason lives in the Pods of the new ReplicaSet, which you inspect with kubectl get pods and kubectl describe pod.

A Deployment reports its health through two conditions in its status. Available is true when at least the required number of replicas (desired minus maxUnavailable) are up and passing readiness - it answers, is the app serving now? Progressing is true while the Deployment is making progress toward the desired state, or has completed it - it answers, is the rollout moving?

The controller does not wait forever. progressDeadlineSeconds - default 600 seconds - is the maximum time a rollout may go without making progress. If no progress happens within that window, the controller flips Progressing to False with the reason ProgressDeadlineExceeded. You will see it in kubectl describe deployment output and in kubectl rollout status, which exits with an error message that the deployment exceeded its progress deadline.

Two properties of this mechanism are exam bait. First, ProgressDeadlineExceeded triggers no automatic action: Kubernetes does not roll back, does not delete the new ReplicaSet, and does not stop retrying the Pods. The condition is a report, not a remedy - recovery is on you, typically via kubectl rollout undo. Second, the two conditions are independent: a Deployment can be Available=True (old Pods still serving) and Progressing=False (new version failed) at the same time. That combination - app up, rollout dead - is precisely the stuck-rollout state from the previous section, now expressed in the API.

Read the conditions before anything else when a delivery misbehaves: they tell you in two lines whether you are facing a stalled rollout, a capacity loss, or both.

Almost every stuck rollout reduces to one cause: the new Pods never become Ready, so the readiness gate never opens. What differs is why, and each why has a distinct fingerprint in kubectl get pods.

Three of these deserve emphasis. A bad image tag is the classic release mistake - a typo in the tag or an image that was never pushed - and it can never self-heal; the kubelet retries the pull with backoff forever. A failing readiness probe is subtler: the containers run, logs may look normal, but the Pod shows 0/1 Ready, endpoints are never updated, and the rollout cannot progress - this is the textbook example of readiness blocking a rollout. Insufficient resources surface as Pending Pods, and note that maxSurge makes a rollout briefly need more capacity than steady state, so a cluster that fits the app may still not fit the app mid-rollout.

In every case the old ReplicaSet remains up, bounded by maxUnavailable. The rollout machinery has already contained the blast radius; your task is diagnosis, then the decision to fix forward or roll back.

When the new version is the problem, the fastest recovery is usually to return to the version that worked. Every time a Deployment's Pod template changes, the controller keeps the previous ReplicaSet - scaled to zero - as a numbered revision. Rollback re-instates an old revision's Pod template:

kubectl rollout history deployment/web
kubectl rollout history deployment/web --revision=3
kubectl rollout undo deployment/web
kubectl rollout undo deployment/web --to-revision=2
kubectl rollout status deployment/web

history lists the revisions, undo returns to the immediately previous one, and --to-revision targets a specific number. A rollback is itself a rolling update - the Deployment scales the old ReplicaSet back up under the same readiness gating - and it creates a new revision number rather than rewinding the counter. Because the old ReplicaSet still exists, rollback usually only needs to start Pods from an image nodes have already pulled, which makes it fast.

Two supporting details matter. revisionHistoryLimit controls how many old ReplicaSets the Deployment retains - the default is 10 - and setting it to 0 deletes history and makes rollout undo impossible, a trade-off question KCNA can pose. And the CHANGE-CAUSE column in rollout history is populated from the kubernetes.io/change-cause annotation if you set it; without it, history shows revision numbers with no explanation, which is why teams that rely on manual rollbacks annotate their deploys.

Finally, remember what undo does not fix: if the rollout is stuck because the cluster lacks capacity or a referenced Secret is missing, rolling back the image will not create nodes or Secrets. Match the remedy to the diagnosis.

Walk through a realistic failure end to end. Your team ships version 2.1.0 of a web API by updating the Deployment's image to shop/api:2.10 - a typo, since the registry only has 2.1.0. The deploy pipeline applies the manifest and runs kubectl rollout status deployment/api, which prints that 1 out of 4 replicas has been updated and then hangs.

You look at the workload:

kubectl get pods -l app=api
NAME                   READY   STATUS             RESTARTS   AGE
api-5f7c9b6d4-x2kqp    0/1     ImagePullBackOff   0          6m
api-7d9f8c5b21-8xj4w   1/1     Running            0          3d
api-7d9f8c5b21-p9z6t   1/1     Running            0          3d
api-7d9f8c5b21-w4m2r   1/1     Running            0          3d

The picture tells the whole story: one new Pod from the new ReplicaSet is stuck in ImagePullBackOff, while three old Pods keep running - the Deployment created one surge Pod, it never became Ready, so the controller never scaled the old ReplicaSet down. Users have noticed nothing. kubectl describe pod api-5f7c9b6d4-x2kqp confirms it in the Events: failed to pull image, manifest for tag 2.10 not found. After ten minutes the Deployment's Progressing condition flips to False with reason ProgressDeadlineExceeded - and nothing else happens, because Kubernetes does not auto-rollback.

Recovery is a choice. Fix forward: push a corrected manifest with shop/api:2.1.0, and the controller replaces the broken ReplicaSet with a healthy one. Or roll back first: kubectl rollout undo deployment/api returns the Deployment to the previous template, the stuck Pod is deleted, and the rollout completes instantly since the old image is already on the nodes. Either way, the lesson generalizes: stuck rollout, old version serving, diagnosis in the new Pods' events.

When the application was installed as a Helm release, debug at the release level first, because Helm tracks its own state independent of any single Deployment. Three commands cover it:

helm status api
helm history api
helm rollback api 4

helm status shows the release's current state and the notes the chart prints. helm history lists every numbered revision with a status per revision: deployed (the current good state), superseded (replaced by a later revision), failed (the upgrade errored), and pending states such as pending-upgrade (an operation that started and never completed, often because the Helm process was killed mid-run). A release stuck in a pending state blocks further upgrades until it is resolved, typically by rolling back.

helm rollback returns the release to a previous revision - like kubectl rollout undo, it creates a new revision recording the rollback rather than erasing history. Two flags shape failure behavior at upgrade time. --wait makes helm upgrade wait until the resources are actually ready before reporting success, instead of returning as soon as manifests are submitted. --atomic goes further: if the upgrade fails or times out, Helm automatically rolls the release back to the previous revision - the one Helm command that does perform an automatic rollback.

Keep the layering straight for the exam: a helm upgrade that submits a Deployment with a bad image will often report success (without --wait) while the underlying rollout is stuck - Helm applied the manifests; Kubernetes could not realize them. Debugging then proceeds exactly as in the previous sections, at the Deployment and Pod level, and you choose helm rollback or a corrected helm upgrade to recover.

A rollout that completes is not the same as a release that works. Every readiness probe can pass while the new version returns HTTP 500 on a code path the probe never exercises, doubles latency, or silently drops a downstream integration. The final stage of delivery debugging is therefore observational: compare the system's signals immediately after the deploy against its baseline before it.

The signals to watch are the standard ones - error rate, latency, and traffic - scoped to the moment of the rollout. A step change in the error-rate graph that aligns with the deploy timestamp is the classic post-release signature. At the recognition level KCNA expects: metrics tell you something regressed, logs from the new Pods tell you what, and kubectl logs deployment/api (or logs filtered to the new ReplicaSet's Pods) is the quick path to the latter. During a rollout, remember that old and new Pods serve side by side, so mixed log output and partially-degraded error rates are expected mid-transition.

Readiness probes are your automated gate during the rollout, and their quality determines how much protection the rollout machinery gives you: a probe that checks a real dependency-touching endpoint blocks far more bad releases than one that returns 200 unconditionally. Beyond probes, this is exactly the gap progressive delivery tools fill - Argo Rollouts can run an analysis against metrics (for example, a Prometheus error-rate query) while a canary receives traffic and abort the release automatically if the query fails.

The delivery debugging loop, end to end: watch the rollout (rollout status), read the conditions, diagnose the new Pods, roll back or fix forward, then verify with metrics and logs that the version now serving is actually healthy.