What to do when a deploy broke production

In this article

1. Before you start
2. Why interviewers ask this
3. A broken deploy from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the DevOps and cloud learning path, which lists what to learn and in what order.

Why interviewers ask this

The cost of a bad answer here is a candidate who sounds like a debugger when the situation calls for an incident responder. Candidates report this scenario ("a release just went out and the site is failing, what do you do?") in DevOps, cloud and site-reliability interviews. It separates people who have shipped to real users from people who have only read about it.

A weak answer starts by reading logs to find the root cause while users keep failing. A strong answer shows an order of work: confirm the deploy is the trigger, stop the damage, verify recovery, then investigate. It also names what a rollback cannot undo, such as data already written. A Staff-level answer goes one step earlier and says how the release process should have limited the harm, with small releases, a canary and a quick way to switch a feature off.

A broken deploy from first principles

Rollback

A rollback is an action that returns a service to the last version known to work, by redeploying the previous build or configuration, so that users stop meeting the new defect before anyone has understood its cause.

Canary release

A canary release is a partial, time-limited deployment of a change to a small share of a service, evaluated against the rest of the service (the control) to decide whether to continue, pause or roll back.

In plain words

Imagine a restaurant that changes a recipe on the menu. Diners start getting sick. The owner does not first study the kitchen; she stops serving the dish, then investigates. A canary is serving the new dish to one table first. A rollback is putting the old recipe back.

The analogy breaks in one place. A restaurant can throw the dish away, but software can have written data that nobody can take back.

In one precise sentence, a deploy broke production when a change you released made an indicator such as the error rate fall far below its target over a short window, burning the error budget much faster than normal, and the response is to stop the damage first and find the cause second.

Step 1: A canary limits how many users a bad deploy reaches

A canary release limits damage because only a small share of users ever receives the change. The Google site reliability engineering (SRE) Workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation". The canary is the part of the service that gets the change, and the control is the rest.

The workbook's own example shows the effect. A defect that causes 20% errors on all traffic is a 20% overall error rate. On a 5% canary the same defect is "a 1% overall error rate". A canary sends a change to a small share of traffic first, so the same bug reaches twenty times fewer users.

Canaries find what tests miss, because test environments are not identical to production and tests do not cover every scenario. Judge the canary on metrics close to user pain, such as error responses and latency, and roll back or page a human if its error rate is too far from the control's. Run one canary at a time.

Step 2: Metrics against the target show that production is broken

Production counts as broken when an indicator such as the error rate falls far below its target over a short window, so the error budget burns much faster than normal; you do not wait for the whole service-level objective (SLO) period to be missed. The SRE Workbook chapter on alerting on SLOs builds its alerts on this burn rate. A service level indicator (SLI) is the ratio of good events to total events, such as successful requests divided by all requests, according to the SRE Workbook. A service level objective (SLO) is a target percentage for that ratio.

The Google workbook calls a release successful only if it is made available to customers with no severe defects or SLO violations. So a deploy broke production when the error rate or latency right after the release burns the budget far faster than normal, and the timing points at the deploy.

Configuration changes count as deploys too: in Google's experience, "a majority of incidents are triggered by binary or configuration pushes". Error and latency metrics against the SLO show that production is broken; a dashboard of processor load alone does not.

Step 3: A rollback or a flag stops the damage before the cause is known

Declare an incident, stop further deploys so nothing stacks on top of the broken release, and say who is coordinating and who is updating stakeholders while you mitigate. Stopping the damage means returning users to a working state while you still do not understand the bug. There are three ways. A rollback reverts to the last good build. A feature flag, a switch in the code that turns one feature on or off without a new build, can disable a feature that was launched behind it. Rolling forward means patching and redeploying.

Without a rollback option, the SRE Workbook says patching and redeploying during the outage will almost certainly prolong the user impact, so a rollback path should exist before you need it.

In Kubernetes, a Deployment keeps updates versioned, and any update can be reverted to a previous stable version. The command is kubectl rollout undo deployment/nginx-deployment, and --to-revision=2 picks a specific earlier revision, as the Deployments page shows.

A rolling update is a gradual replacement of old Pods with new ones. The same gradual replacement happens on the way back, so the undo takes time. It also reverts only what the Deployment controls, so data already written stays written.

Step 4: A review and failure metrics turn one incident into learning

After users are safe, find the root cause, write down what happened without blaming a person, and ask what would have limited the harm. Review the incident and track the change failure rate, which is the percentage of deployments that need a hotfix or a rollback. DevOps Research and Assessment (DORA) describes it as a proxy for the amount of rework a team must perform.

DORA also tracks failed deployment recovery time, the time to recover from a failed deployment. The old metric "mean time to recover (MTTR)" was renamed and redefined as this in 2023 (DORA), and it covers only failures caused by a change. DORA's current model groups change fail rate with deployment rework rate as its two instability measures.

Smaller releases help here. The SRE Workbook says smaller, self-contained release artifacts make it cheaper and easier to roll back any given artifact. DORA's research also debunked the myth that speed comes at the expense of stability, because high performers excel at both.

1 · Canary2 · Detect3 · Stop the damage4 · LearnReplay

Step-by-step description

1. A small share of traffic gets the change first.
2. Error and latency metrics show the break.
3. Roll back or switch the flag off.
4. Review the incident and track failures.

• A canary sends a change to a small share of traffic first.
• Error and latency metrics against the SLO show that production is broken.
• A rollback or a feature flag stops the damage before the cause is known.
• Review the incident and track the change failure rate.

Worked example

The numbers below are illustrative traffic with the shape of the SRE Workbook's own example: a 99.9% success target over 3,000,000 requests in four weeks gives 3,000 allowed errors, and a single outage with 1,500 errors costs 50% of that budget. Spread evenly, that traffic is about 1.24 requests per second.¹

Now suppose a release makes 20% of requests fail. If it goes to everyone and someone rolls it back after 10 minutes, the service serves about 149 errors, or 4.96% of the budget. If the same defect sits on a full rollout for 60 minutes, it serves about 893 errors, or 29.76% of the budget.¹ A 5% canary running the same defect for 10 minutes serves about 7 errors, or 0.25% of the budget, and for 60 minutes about 45 errors, or 1.49%.¹

The assumption is uniform load, as in the workbook; real traffic peaks, so a deploy during a peak costs more than the average suggests.

Both levers matter. The full rollout caught after 60 minutes uses 29.76% of the budget and a 5% canary caught after 10 minutes uses 0.25%.¹ Even the 60-minute full rollout leaves the four-week target met, yet it is an incident, because the budget burns far faster than normal. Exposure and detection time multiply, so a small canary with fast alerts is worth far more than either alone.

Text description of the diagram

• The full rollout caught after 60 minutes uses 29.76% of the budget.
• A 5% canary caught after 10 minutes uses 0.25% of the budget.

Two smaller calculations from the same script. In a Kubernetes Deployment with 4 replicas and the default of 25% for how many Pods may be unavailable, at most 1 Pod is down at a time during an update (illustrative replica count).¹ And if 2 of 10 deploys in a week needed a hotfix or a rollback, the change failure rate is 20% (illustrative).¹

Trade-offs and edge cases

Each way of stopping the damage trades speed against reach, and none of them is free.

Text description of the diagram

• A rollback reverts the build but not data changes.
• A feature flag switches off one feature without a new build.
• Rolling forward prolongs user impact when no rollback exists.

A rollback reverts the build but not data changes. A feature flag switches off one feature without a new build, but the flag must exist before the incident. Rolling forward prolongs user impact when no rollback exists, because you must find, patch and deploy a fix during the outage, as the SRE Workbook notes. Decide in order: roll back if a known good version exists, switch a flag off if one covers the change, and roll forward last.

Edge cases an interviewer may probe:

• Data changes. A rollback reverts what the Deployment controls. A database change that already ran, or a changed setting stored elsewhere, stays in place, so a safe release keeps the old and new code compatible with the data during the transition.
• Configuration pushes. They trigger incidents too: Google reports that most of its incidents follow a binary or configuration push, so version and canary configuration like code.
• Off-peak canaries. The SRE Workbook warns that deploying off-peak can miss defects that only appear under load.
• Several changes in flight. One canary at a time keeps the cause clear; overlapping canaries make a bad signal impossible to attribute.
• Tiny canaries. A canary too small to see an error rate gives no evidence, so size it to carry enough traffic for the metric you judge it on.

The Kubernetes interview questions page covers the Deployment mechanics in more depth, and the Amazon Web Services (AWS) questions for DevOps page covers the same ideas on a cloud platform.

How to answer it in an interview

The spoken structure: confirm, stop the damage, verify recovery, then investigate, and finish with how you would limit the harm next time.

Common mistakes

• "I would start by reading the logs to find the bug." (wrong) Users keep failing while you search. (right) "I stop the damage first with a rollback or a flag, then investigate."
• "Rollback fixes everything." (wrong) It reverts the deployed build, not data already written. (right) "I roll back, then check whether data or settings need their own repair."
• "The tests passed, so it was safe." (wrong) Test environments are not identical to production and tests do not cover every scenario. (right) "Tests reduce risk, and a canary catches what they miss."
• "We should deploy less often so nothing breaks." (wrong) Larger releases are harder to diagnose and roll back, and DORA found high performers are fast and stable. (right) "I ship small changes behind a canary."
• "I would watch processor load on the canary." (wrong) Resource graphs are weakly tied to user pain. (right) "I judge the canary on error responses and latency."

Follow-up questions

How do you decide between rollback and roll forward?

Roll back when a known good version exists and the new one is clearly the trigger, because it is the fastest known fix. Roll forward only when the old version cannot run, for example when a data change is not reversible, and treat the patch as higher risk.

How do you roll back a Kubernetes Deployment?

Run kubectl rollout undo deployment/<name> to return to the previous revision, or add --to-revision=<n> for a specific one, as the Kubernetes documentation shows. Then watch the metrics, since the undo is a gradual replacement and not a switch.

Does rolling back a deploy undo a database change?

Not by itself, because a rollback of the deployed build does not run the reverse of a data change. Plan changes so the old code still works with the new data, and treat the data fix as a separate step.

How do you measure how often deploys fail?

Use the change failure rate: failed deployments divided by total deployments, where failed means needing a hotfix or a rollback, as DORA describes. Pair it with failed deployment recovery time, which tells you how fast you restore service. Both depend on a reliable CI/CD pipeline that can ship and revert quickly.

Why not deploy less often to be safe?

Bigger releases bundle more changes, so each failure is harder to diagnose and costlier to undo. Smaller, frequent releases limit the blast radius. The git commands and branching habits that keep changes small are part of that safety.

Sources

1. Canarying Releases (SRE Workbook, chapter 16) — Google SRE, 1 Jul 2018
2. Implementing SLOs (SRE Workbook, chapter 2) — Google SRE, 1 Jul 2018
3. Alerting on SLOs (SRE Workbook, chapter 5) — Google SRE, 1 Jul 2018
4. Deployments — Kubernetes documentation, 7 Jul 2026
5. Performing a Rolling Update — Kubernetes documentation, 26 Oct 2026
6. A history of DORA's software delivery metrics — DORA (Google Cloud), 2 Jan 2026