In this article
Before you start
This page belongs to the site reliability engineering (SRE) interview guide and builds on the DevOps and cloud basics.
Unfamiliar terms are explained in the interview glossary.
Why interviewers ask this
The consequence of a weak answer is easy to picture: a candidate who starts hunting for the root cause while customers are still failing sounds like someone who would extend an outage.
A strong answer shows an ordered habit. You confirm the problem is real, say who owns it, reduce the harm, tell the people affected, check that the fix worked and promise a review. Interviewers also listen for questions you ask back, because a real responder rarely has the full picture. A Staff-level answer adds judgement: when to escalate, when to stop investigating and roll back, and how the follow-up review changes the system rather than the people.
Incident response from first principles
- Incident
- An incident is an unplanned event that harms, or is about to harm, a service's users or business, and that needs a coordinated response rather than routine work. A failing checkout is an incident; a slow nightly report that nobody waits for usually is not.
- Mitigation
- Mitigation is an action that reduces the harm of an incident without necessarily fixing its cause, such as a rollback, turning off a feature or moving traffic away from a broken region. It is judged by whether users stop being hurt.
In plain words
Think of a kitchen fire. First you shout that there is a fire and make sure one person is in charge. Then you smother the flames or cut the gas. Only after the room is safe do you investigate why the pan ignited. The analogy breaks in one place: software failures often have no visible flames, so you must choose a signal that tells you the fire is out.
In one precise sentence, incident response is a repeatable order of actions that reduces user harm quickly, keeps people coordinated and leaves behind a lesson. The structure follows the practice in Google's chapter on managing incidents and its incident response workbook chapter. PagerDuty's public incident documentation shows a vendor's version of the same roles. The four steps below are the author's own summary of that common shape, not a quoted standard.
Step 1: Declare early and name an owner
Declare the incident early and name one person in charge. In the interview, say the words "I would declare an incident" and name the incident commander, even if the person is you. Declaring costs little: it opens a channel, starts a timeline and tells others to expect updates. Waiting until you are sure costs more, because the harm continues while you decide.
Input: an alert or a report. Output: an incident with an owner and a shared place to talk. Larger responses split work into roles such as a commander, someone who operates the system and someone who communicates; Google's managing incidents chapter describes this separation. You now know who decides, which makes the next step possible.
Step 2: Triage the impact and ask questions
Triage means judging how bad the incident is and who it touches. Ask back: which users, which feature, since when, and did anything change at that time, such as a deploy or a configuration edit. A severity label, a short rating of how bad the incident is, then sets who gets paged and how often updates go out. Labels differ between companies, so say that you would follow the team's own scale.
Input: first facts. Output: a severity and a hypothesis list ordered by what changed recently. Do not claim a cause here. The goal is a working guess that suggests a safe first action. The SRE guide lists the signals, such as error rate and latency, that you would check first, and monitoring vs logging vs tracing explains which tool answers which question.
Step 3: Mitigate before you find the root cause
Mitigate to stop the harm before you hunt for the root cause. If errors began when a new version shipped, roll it back; if one feature is failing, switch it off; if one region is unhealthy, shift traffic away. Each action is picked because it is quick and reversible, not because it explains anything.
Input: the hypothesis list. Output: users no longer hurt, with the cause still unknown. The root cause can wait because the evidence usually survives in logs and metrics, while customers do not wait. If it costs only seconds, capture a quick snapshot first, such as recent logs, a heap dump or the id of the bad version, so the rollback does not erase clues. Say aloud the one-line reason for your action, for example "the errors started with the deploy, so rolling back is the fastest safe test".
Step 4: Verify recovery before you stand down
Verify recovery with the same signal that raised the alert. If an error-rate graph fired the page, the incident is not over until that graph returns to its normal band and stays there for a stated time. "The rollback finished" is an action, not proof.
Input: the mitigation. Output: evidence that users are fine, then a closing message and a promise of a review. Candidates often stop at the mitigation; saying how you would check, and for how long, is the rubric line that moves an answer up a level. Then schedule a blameless review; the blameless postmortem interview answer page shows how to speak about it. A blameless postmortem is a written review of an incident that explains what happened and what to change while treating mistakes as flaws in the system, not faults of individuals; the idea is covered in Google's chapter on postmortem culture.
Step-by-step description
- Name one owner.
- Judge who is affected.
- Stop the harm first.
- Check the alert signal.
- Declare the incident early and name one person in charge.
- Mitigate to stop the harm before you hunt for the root cause.
- Verify recovery with the same signal that raised the alert.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take an illustrative case. A new version of a checkout service is deployed at minute 0, and 38% of requests start failing out of 1,200 requests a minute.1 An alert fires at minute 4, the on-call engineer declares the incident at minute 9 and names themself commander, a rollback begins and the error rate falls back to normal at minute 21, and the engineer confirms the graph stayed normal until the thirty-third minute.1
For the first 21 minutes, until the mitigation landed, users were affected, which is 9,576 failed requests.1 Declaring ten minutes later would have added 4,560 failed requests, assuming mitigation also slipped by ten minutes, a cost that no amount of later investigation recovers.1 None of these numbers are measurements; they exist so that you can see how the order of actions changes the outcome.
| Quantity | Value (illustrative, computed) |
|---|---|
| Deploy to alert | 4 minutes |
| Alert to declaration | 5 minutes |
| Declaration to mitigation | 12 minutes |
| Customer impact before mitigation | 21 minutes |
| Mitigation to verified recovery | 12 minutes |
| Failed requests before mitigation | 9,576 |
| Extra failures if declared ten minutes later | 4,560 |
Text description of the diagram
- Customers were affected for 21 minutes before the mitigation landed.
- Declaring ten minutes later would have added 4,560 failed requests.
Trade-offs and edge cases
The fixed order has exceptions. Rolling back is quick only when the change is reversible; a database migration may not be, and then you may need to fix forward, which means shipping a new change to repair the problem. Declare early also has a cost: too many declared incidents train people to ignore them, so teams define thresholds, and you should say you would follow the team's.
If you are the only person on call, you play every role; the on-call culture question for managers covers the team side of this. Say so, name the roles as you switch, and ask for help early, because a second pair of eyes catches mistakes made under pressure. If the cause is unclear, treat recent changes as suspects first, since they are the cheapest hypotheses to test.
Security incidents follow a different framework. For roles on a security team, the reference is the National Institute of Standards and Technology (NIST) publication 800-61 Rev. 3 (April 2025), which organises incident response around the functions of the Cybersecurity Framework (CSF) 2.0: Govern, Identify, Protect, Detect, Respond and Recover. It replaced Rev. 2 (2012), whose four-phase lifecycle still appears in many interviews: preparation; detection and analysis; containment, eradication and recovery; and post-incident activity.
A security interviewer listens for containment before eradication, evidence preserved with a chain of custody, and escalation to legal or communications. The structure and terms differ from the operations practice above, so confirm which kind of role the interviewer means before choosing a framework. A rollback is rarely the first move there. A drop in a machine learning model's quality in production is handled in the same declare, mitigate, verify order, for example by reverting to the previous model version. The interview preparation library lists the neighbouring topics.
How to answer it in an interview
Checkout errors just spiked in production and you are on call. What do you do?
I would confirm the alert is real and see how many users it touches, then declare an incident and name one owner, because a named owner stops people running conflicting commands. I would check what changed at the time the errors began. If a deploy matches, I would roll it back before looking for the cause, and have someone post short updates. I would then confirm the error rate returns to normal on the same graph that alerted. Afterwards I would write a blameless review covering what we change.
The spoken structure: declare and name an owner, triage with a question about recent changes, mitigate with a reason, verify with the original signal, then promise the review.
Common mistakes
A weak answer and a strong answer differ in four places, shown below.
Text description of the diagram
- A weak answer jumps to the root cause.
- A strong answer states the first action and why it is safe.
A weak answer jumps to the root cause. A strong answer states the first action and why it is safe.
- "I would dig through the logs until I find the bug." (wrong) The users keep failing while you read. (right) "I would first reduce the harm with a rollback, then use the logs to find the cause."
- "I would fix it myself and tell people afterwards." (wrong) Silence leaves others to duplicate effort or to guess. (right) "I would name an owner and send short updates on a regular rhythm."
- "The rollback is done, so we are finished." (wrong) An action is not evidence. (right) "I would watch the alerting signal until it stays normal for an agreed time."
- "Whoever shipped the change caused it." (wrong) That turns the review into blame, and people hide mistakes. (right) "I would ask which gaps let the change reach users and what we add to close them."
Follow-up questions
What are typical incident response interview questions?
Expect scenario prompts such as a latency spike, a failed deploy or a full disk, plus questions candidates report on how you set severity, how often you update stakeholders, when you escalate and to describe an incident you handled. The same four steps answer all of them; the SRE interview guide collects the wider topic list.
How is incident response for security roles different?
Security incidents are organised around the CSF 2.0 functions in NIST publication 800-61 Rev. 3 (April 2025), where containing an attacker and preserving evidence come before restoring service. Operations incidents usually prioritise restoring service quickly. Ask the interviewer which kind they mean.
What is a blameless postmortem?
A blameless postmortem answer is your chance to show that you fix systems, not people. Interviewers want to hear that you would turn each contributing cause into a follow-up action with an owner, such as adding a check or an alert, rather than warning an individual.
Should you roll back or fix forward?
Roll back when the change is recent, reversible and the likely cause, because it is the fastest safe action. Fix forward when a rollback is impossible or would cause its own harm, as with some data migrations.
What if you are the only person on call?
Take every role in turn, say which one you are playing, and escalate early. Declaring an incident still helps, because it creates a record and prompts others to join.
How do you prepare for a broader DevOps round?
Practise explaining your reasoning aloud, not only the commands; the same four steps apply to most scenario prompts. The DevOps and cloud preparation guide sets out what to learn first, and the site reliability engineer guide covers the incident topics in context.
- Declare early and name one owner before you investigate.
- Triage by asking who is affected and what changed.
- Mitigate to stop harm first; the root cause can wait.
- Verify recovery with the same signal that raised the alert.
- Close with a blameless review that changes the system.
Sources
- Managing Incidents (Site Reliability Engineering, Google) — Google SRE, 1 Jan 2017
- Incident Response (The Site Reliability Workbook, Google) — Google SRE, 27 Jul 2018
- Postmortem Culture: Learning from Failure (Google) — Google SRE, 1 Jan 2017
- Incident Response Recommendations and Considerations for Cybersecurity Risk Management (NIST publication 800-61 Rev. 3) — NIST Computer Security Resource Center, 3 Apr 2025
- PagerDuty Incident Response documentation — PagerDuty, 26 Oct 2026