Debugging Application Delivery: Stuck Rollouts, Failed Releases, and Rollbacks

Generic Pod troubleshooting asks why a container is crashing. Delivery debugging asks a different question: you shipped a new version - did it arrive, and is it serving? The failure modes are specific to the rollout machinery: a new ReplicaSet that never becomes healthy, a rollout frozen halfway, a Helm upgrade stuck in a failed state, or a release that technically completed but doubled the error rate. The KCNA delivery domain probes this distinction, so keep the two mindsets separate.

The single most important fact is that a rolling update is gated on readiness. The Deployment controller creates a new ReplicaSet and scales it up in steps, but it only continues - and only scales the old ReplicaSet down - as new Pods pass their readiness probes and count as available. The bounds maxSurge and maxUnavailable guarantee a minimum serving capacity at every step.

Both default to 25% of the desired replicas, and they round in opposite directions: maxSurge rounds up (extra Pods allowed above the desired count) and maxUnavailable rounds down (Pods allowed below it). With 4 replicas that means at most 5 Pods in total and at least 3 available. You can use this to predict where a broken rollout stops: the controller adds 1 surge Pod and removes 1 old Pod (old Pods are still the available ones, so 3 remain), then fills the freed slot with a second new Pod. If the new Pods never become Ready, the rollout freezes at 3 old + 2 new - no more old Pods can go without dropping below 3 available, and no more new Pods fit under the 5-Pod ceiling.

That gating is what makes delivery failures survivable. If the new Pods never become Ready, the rollout simply stops making progress: the new ReplicaSet sits with a few broken Pods, and the old ReplicaSet stays up and keeps serving traffic. Users are usually unaffected by a completely broken release; what you have is a stuck rollout, not an outage. Your job as the debugger is to notice the hang, find out why the new Pods are not Ready, and either fix forward or roll back. The rest of this lesson is that workflow, in order.

The first tool is kubectl rollout status deployment/web. It blocks and prints progress while the rollout proceeds, exits successfully when the Deployment reaches its desired state, and simply keeps waiting - printing something like Waiting for deployment web rollout to finish: 1 out of 3 new replicas have been updated... - when the rollout is stuck. A status command that never returns is your first signal that the new version is not becoming healthy.

Next, look at the Deployment summary and its ReplicaSets:

kubectl get deployment web
kubectl get replicasets
kubectl describe deployment web

kubectl get deployment shows the columns READY, UP-TO-DATE, and AVAILABLE. During a healthy rollout UP-TO-DATE climbs to the desired count and AVAILABLE follows; in a stuck rollout UP-TO-DATE stalls at a low number while AVAILABLE reflects the old Pods still serving. kubectl get replicasets makes the mechanism visible: mid-rollout you will see two ReplicaSets for the Deployment - the old one still holding most replicas and the new one with a small number that refuse to become ready. That two-ReplicaSet picture is the signature of a stalled rolling update and a favorite exam scenario.

kubectl describe deployment adds the events history - each scale-up and scale-down decision the controller made - and the conditions block covered in the next section. From there the trail leads downward: the Deployment tells you the rollout is stuck, but the reason lives in the Pods of the new ReplicaSet, which you inspect with kubectl get pods and kubectl describe pod.

A Deployment reports its health through conditions in its status, and three condition types matter for delivery debugging:

ReplicaFailure is easy to forget because it only appears when something goes wrong; a healthy Deployment shows just Available and Progressing. Note that ProgressDeadlineExceeded is a reason on the Progressing condition, not a condition type of its own, and condition types such as Initialized or ContainersReady belong to Pods, not Deployments.

The controller does not wait forever. progressDeadlineSeconds - default 600 seconds - is the maximum time a rollout may go without making progress. The clock is not measured from the start of the rollout: every sign of progress - a new ReplicaSet being created or scaled, a new Pod becoming available - resets it. A slow rollout that brings up one Pod every few minutes never trips the deadline, while a rollout whose new Pods never become available trips it 600 seconds after its last step forward. When the window passes without progress, the controller sets Progressing to False with the reason ProgressDeadlineExceeded. You will see it in kubectl describe deployment output and in kubectl rollout status, which exits with an error message that the deployment exceeded its progress deadline.

Two properties of this mechanism are exam bait. First, ProgressDeadlineExceeded triggers no automatic action: Kubernetes does not roll back, does not delete the new ReplicaSet, and does not stop retrying the Pods. The condition is a report, not a remedy - recovery is on you (or on automation you add that watches the condition), typically via kubectl rollout undo or a corrected manifest. Second, the conditions are independent: a Deployment can be Available=True (old Pods still serving) and Progressing=False (new version failed) at the same time. That combination - app up, rollout dead - is precisely the stuck-rollout state from the previous section, now expressed in the API.

Read the conditions before anything else when a delivery misbehaves: they tell you in a few lines whether you are facing a stalled rollout, a capacity loss, a Pod-creation rejection, or a combination.

Almost every stuck rollout reduces to one cause: the new Pods never become Ready - or are never created - so the readiness gate never opens. What differs is why, and each why has a distinct fingerprint.

A bad image tag is the classic release mistake - a typo in the tag or an image that was never pushed - and it can never self-heal; the kubelet retries the pull with backoff forever. A failing readiness probe is subtler: the containers run, logs may look normal, but the Pod shows 0/1 Ready, it is never added to the Service's endpoints, and the rollout cannot progress - the textbook example of readiness blocking a rollout.

Capacity problems come in three shapes worth separating. Surge needs headroom: maxSurge makes a rollout briefly need more capacity than steady state, so a cluster that fits the app may not fit it mid-rollout, and the surge Pods sit Pending. A Pod whose request is larger than any single node's allocatable capacity stays Pending forever whatever the surge settings - lowering maxSurge or waiting for other Pods to finish does not help; only larger nodes or a smaller request do. And a ResourceQuota rejection is different again: the API server refuses to create the Pod, so there is no Pending Pod to describe. The evidence lives one level up, as a FailedCreate event on the new ReplicaSet (kubectl describe replicaset) and a ReplicaFailure condition on the Deployment.

In every case the old ReplicaSet remains up, bounded by maxUnavailable. The rollout machinery has already contained the blast radius; your task is diagnosis, then the decision to fix forward or roll back.

Kubernetes has three probe types, and each produces a different symptom when a new version misbehaves. Only one of them - readiness - is what a rolling update waits on.

The quick tell between the first two is the RESTARTS column: a readiness failure leaves the count at zero, a liveness failure makes it climb. A startupProbe exists for applications that take a long time to boot; it protects them from a liveness probe that would otherwise kill them mid-start. It says nothing about whether the app is healthy afterwards, so it cannot be what gates a rollout on real health - that remains the readiness probe's job.

Because the readiness probe is the gate, its design decides how much protection a rollout gives you. A probe that exercises the real serving path stops more bad releases than one that returns 200 unconditionally. There is a caveat, though: if readiness also checks a shared dependency such as a database, an outage of that dependency marks every Pod - old and new - not Ready at once, and the Service is left with no endpoints. Check what the Pod itself needs to serve; leave shared-dependency failures to alerting and graceful error handling.

When the new version is the problem, the fastest recovery is usually to return to the version that worked. Every time a Deployment's Pod template changes, the controller keeps the previous ReplicaSet - scaled to zero - as a numbered revision. Rollback re-instates an old revision's Pod template:

kubectl rollout history deployment/web
kubectl rollout history deployment/web --revision=3
kubectl rollout undo deployment/web
kubectl rollout undo deployment/web --to-revision=2
kubectl rollout status deployment/web

The two revision flags look alike but do opposite kinds of work:

A rollback is itself a rolling update - the Deployment scales the old ReplicaSet back up under the same readiness gating - and it creates a new revision number rather than rewinding the counter. Because the old ReplicaSet still exists, rollback usually only needs to start Pods from an image nodes have already pulled, which makes it fast.

revisionHistoryLimit controls how many old ReplicaSets the Deployment retains - the default is 10 - so --to-revision can reach back across any of those retained revisions, not just the previous one. Setting it to 0 deletes history and makes rollout undo impossible, a trade-off question KCNA can pose. (The long-removed extensions/v1beta1 Deployment API kept every old ReplicaSet by default; current apps/v1 Deployments default to 10.)

The CHANGE-CAUSE column in rollout history is read from one annotation only, and several similar-looking annotations do other jobs:

Without a change-cause, history shows revision numbers with no explanation, which is why teams that rely on manual rollbacks annotate their deploys.

Finally, match the remedy to the diagnosis. If the rollout is stuck because the cluster lacks capacity, a quota is exhausted, or a referenced ConfigMap or Secret is missing, rolling back the image will not create nodes, quota or objects - though returning to a template that does not need them can. Recreating Pods does not help either: deleting Pods or running kubectl rollout restart starts fresh Pods from the same image and the same references (restart only stamps a restart timestamp on the template), so a bad tag or a missing ConfigMap fails again exactly as before. The fix is a changed template - a corrected manifest, or a rollback to a revision that worked.

Walk through a realistic failure end to end. Your team ships version 2.1.0 of a web API by updating the Deployment's image to shop/api:2.10 - a typo, since the registry only has 2.1.0. The deploy pipeline applies the manifest and runs kubectl rollout status deployment/api, which prints that 2 out of 4 new replicas have been updated and then hangs.

You look at the workload:

kubectl get pods -l app=api
NAME                   READY   STATUS             RESTARTS   AGE
api-5f7c9b6d4-x2kqp    0/1     ImagePullBackOff   0          6m
api-5f7c9b6d4-k7d2n    0/1     ImagePullBackOff   0          6m
api-7d9f8c5b21-8xj4w   1/1     Running            0          3d
api-7d9f8c5b21-p9z6t   1/1     Running            0          3d
api-7d9f8c5b21-w4m2r   1/1     Running            0          3d

The picture tells the whole story: two new Pods from the new ReplicaSet are stuck in ImagePullBackOff, while three old Pods keep running. That is exactly where the default 25% bounds freeze a 4-replica rollout: one surge Pod, one old Pod removed, one more new Pod in its place - and since neither new Pod becomes Ready, the controller can remove no further old Pods. Users have noticed nothing. kubectl describe pod api-5f7c9b6d4-x2kqp confirms it in the Events: failed to pull image, manifest for tag 2.10 not found. Ten minutes after the last progress, the Deployment's Progressing condition flips to False with reason ProgressDeadlineExceeded - and nothing else happens, because Kubernetes does not auto-rollback.

Recovery is a choice. Fix forward: push a corrected manifest with shop/api:2.1.0, and the controller replaces the broken ReplicaSet with a healthy one. Or roll back first: kubectl rollout undo deployment/api returns the Deployment to the previous template, the stuck Pods are deleted, and the rollout completes quickly since the old image is already on the nodes. Restarting the Pods would achieve nothing - they would pull the same missing tag. Either way, the lesson generalizes: stuck rollout, old version serving, diagnosis in the new Pods' events.

When the application was installed as a Helm release, debug at the release level first, because Helm tracks its own state independent of any single Deployment. Three commands cover it:

helm status api
helm history api
helm rollback api 4

helm status shows the release's current state and the notes the chart prints. helm history lists every numbered revision with a status per revision: deployed (the current good state), superseded (replaced by a later revision), failed (the operation errored), and pending states such as pending-upgrade (an operation that started and never completed, often because the Helm process was killed mid-run). A release stuck in a pending state blocks further upgrades until it is resolved, typically by rolling back.

helm rollback returns the release to a previous revision - like kubectl rollout undo, it creates a new revision recording the rollback rather than erasing history.

Two flags shape failure behaviour, and Helm 4 (the current major version) changed both:

What the automatic undo does depends on the operation. On a failed upgrade, Helm rolls the release back to the previous successful revision. On a failed first install there is no earlier revision to return to, so the release is uninstalled instead. Apart from progressive-delivery tools such as Argo Rollouts (covered below), this is the only automatic rollback in the toolchain covered here - the Kubernetes Deployment controller never does one.

Keep the layering straight: a helm upgrade that submits a Deployment with a bad image will often report success when it is not waiting for resource readiness - Helm applied the manifests; Kubernetes could not realize them. Debugging then proceeds exactly as in the previous sections, at the Deployment and Pod level, and you choose helm rollback or a corrected helm upgrade to recover.

A rollout that completes is not the same as a release that works. Every readiness probe can pass while the new version returns HTTP 500 on a code path the probe never exercises, doubles latency, or silently drops a downstream integration. The final stage of delivery debugging is therefore observational: compare the system's signals immediately after the deploy against its baseline before it.

The signals to watch are the standard ones - error rate, latency, and traffic - scoped to the moment of the rollout. A step change in the error-rate graph that aligns with the deploy timestamp is the classic post-release signature. At the recognition level KCNA expects: metrics tell you something regressed, logs from the new Pods tell you what, and kubectl logs deployment/api (or logs filtered to the new ReplicaSet's Pods) is the quick path to the latter. During a rollout, remember that old and new Pods serve side by side, so mixed log output and partially-degraded error rates are expected mid-transition.

Readiness probes are your automated gate during the rollout, but they only see what a single Pod reports about itself. A regression in error rate is a fleet-level metric, and it is worth being precise about which tools can act on it:

So if a release must be stopped with no one watching, the answer is a metric-gated rollout, not an alert or an autoscaler.

The delivery debugging loop, end to end: watch the rollout (rollout status), read the conditions, diagnose the new Pods, roll back or fix forward, then verify with metrics and logs that the version now serving is actually healthy.