In this article
- The vocabulary of site reliability engineering
- What SRE rounds test
- Start here: the learning path
- Reliability targets: SLIs, SLOs and error budgets
- Monitoring: golden signals and symptoms
- Incident response: the first thirty minutes
- Capacity planning and on-call
- Safe change: releases, rollbacks and canaries
- Toil and blameless postmortems
- Worked example: when to page on a burn rate
- Questions interviewers ask on SRE
- All 5 guides in Site reliability (SRE)
- Follow-up questions
- Sources
The vocabulary of site reliability engineering
- Site reliability engineering
- Site reliability engineering (SRE) is the practice of running production services with software engineering methods, so that availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity are handled by code and measurement instead of manual effort. The Google book on the subject puts it as what happens when you ask a software engineer to design an operations team.
- Service level indicator
- A service level indicator (SLI) is a carefully defined quantitative measure of some aspect of the level of service provided, such as the fraction of requests that succeed or the latency of a request. It is the thing you measure, and a service level objective is the target you set for it.
- Toil
- Toil is a kind of operational work, tied to running a service, that is manual, repetitive, automatable, tactical and devoid of enduring value, and that grows in step with the size of the service. The SRE book sets a goal of keeping it below half of each engineer's time.
- Error budget
- An error budget is the unreliability a service may spend in a period, equal to one minus its objective; a 99.9 percent objective over 30 days leaves 43.2 minutes1.
- Burn rate
- Burn rate is a ratio that says how fast, relative to the service level objective, a service consumes its error budget. A burn rate of 1 uses the whole budget by the end of the window; a burn rate of 14.4 uses it 14.4 times faster.
What SRE rounds test
An SRE loop tests judgement under constraints: can you say how reliable a service should be, notice when it is not, respond when it breaks, plan for growth, change it without breaking it, and learn from the failure. We group the questions into six areas, one section each, and the learning path follows the order a service lives through them.
Text description of the diagram
- The six areas are reliability targets, monitoring, incident response, capacity and on-call, safe change, and toil and postmortems.
- The learning path follows the order a service lives through them.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Start here: the learning path
Take the areas in this order. Each section below gives the core of one area.
- Reliability targets: service level indicators, service level objectives (SLOs) and error budgets.
- Monitoring: golden signals, symptoms versus causes.
- Incident response: the first thirty minutes.
- Capacity planning and on-call.
- Safe change: releases, rollbacks and canaries.
- Toil and blameless postmortems.
- How do you answer an incident response scenario interview?
- How do you answer capacity planning and on-call questions?
- Monitoring vs logging vs tracing: what is the difference?
Reliability targets: SLIs, SLOs and error budgets
A reliability target is stated as a service level objective, a target value for a service level measured by an indicator, and the gap between the target and perfect is the error budget. The Google SRE book defines a service level objective as "a target value or range of values for a service level that is measured by an SLI" (Service Level Objectives, 2017).
The same chapter lists request latency, error rate, throughput, availability and durability as common indicators, and prefers percentiles to averages, because an average hides slow requests. Teams jointly define a quarterly error budget from the objective, and when violations use it up, "releases are temporarily halted" while more is invested in testing and development (Embracing Risk, 2017). System design shows where these targets enter a design, and DevOps and cloud shows error budgets inside a release process.
The arithmetic is the interview staple. A 99.9 percent objective over 30 days allows 43.2 minutes of downtime; 99.99 percent allows 4.32; 99.999 percent allows 0.4321. The book adds that a 99.99 percent target allows 52.56 minutes a year and that "100% is probably never the right reliability target" because it is more reliability than users notice (Embracing Risk, 2017).
Text description of the diagram
- A 99.9 percent objective over 30 days allows 43.2 minutes of downtime.
- A 99.99 percent objective allows 4.32 minutes and 99.999 percent allows 0.432 minutes.
Monitoring: golden signals and symptoms
Monitoring answers two questions, what is broken and why, and the SRE book's rule is to page on symptoms and to alert on causes only when they are very definite and very imminent (Monitoring Distributed Systems, 2017). It names four measures called golden signals: latency, the time to serve a request; traffic, the demand placed on the system; errors, the rate of failing requests; and saturation, how full the service is.
It also separates white-box monitoring, based on metrics exposed by the system's internals, from black-box monitoring, which tests externally visible behavior as a user would see it. A strong interview answer picks one signal per failure story: a rise in latency with flat traffic points at saturation or a slow dependency.
Incident response: the first thirty minutes
Incident response is the discipline of restoring service first and finding the cause second. The usual interview scenario gives you an alert and a vague symptom and watches your order of moves: confirm the impact and its scope, check what changed recently, mitigate by rolling back or shifting traffic, communicate on a schedule, and only then dig for the root cause.
The postmortem triggers in the SRE book show which incidents need a written review: user-visible downtime above a threshold, data loss of any kind, an on-call engineer's intervention such as a rollback or a traffic reroute, a long resolution time, or a monitoring failure (Postmortem Culture, 2017). The candidate who says "I would find the root cause first" while the service is down loses the point.
Capacity planning and on-call
Capacity planning is estimating the demand a service will face and making sure the resources exist before it does, and on-call is the rota under which an engineer is available to respond to alerts. The SRE book lists capacity planning among a team's core responsibilities and says Google caps the aggregate operations work, which includes tickets, on-call and manual tasks, at 50 percent of SRE time (Introduction, 2017). Interview versions combine arithmetic and judgement: size the peak, add headroom, and say what happens when a quarter of capacity fails. A strong answer states the peak in requests per second, the headroom as a percentage, and the replica count that gives it. The sizing skill is taught in back-of-the-envelope estimation.
Safe change: releases, rollbacks and canaries
Safe change is releasing a new version so that a bad one is noticed and undone before it spends the error budget, and it matters because the SRE book says roughly 70 percent of outages are due to changes in a live system (Introduction, 2017). The error budget gives the policy: when budget remains, release; when it is spent, slow down.
A canary release sends a small share of traffic to the new version first, and a rollback restores the previous one. The release strategies, with their trade-offs, are covered in deployment strategies, and the container platform questions in Kubernetes interview questions. In an interview, say how you would notice a bad release (the same symptom alerts as above), how fast you can undo it, and what the blast radius is if you cannot.
Toil and blameless postmortems
Toil is work that a machine could do and that leaves nothing behind, and a postmortem is how an incident leaves something behind. The toil chapter of the SRE book describes it as manual, repetitive, automatable, tactical, without enduring value and scaling linearly with the service, and sets a goal of under 50 percent of each engineer's time (Eliminating Toil, 2017). The postmortem chapter says a blameless postmortem focuses on contributing causes "without indicting any individual or team". Expect the behavioural version of both: "tell me about toil you removed" and "tell me about an incident you were part of". Prepare one toil story and two incident stories, each with numbers, using the structure in the behavioural interviews guide.
Worked example: when to page on a burn rate
A burn-rate alert pages when the error budget is being spent too fast, and the Site Reliability Workbook gives parameters for a 99.9 percent objective over 30 days (Alerting on SLOs, 2018). The budget is 43.2 minutes1. If the observed error ratio is 1.44 percent against an allowed 0.1 percent, the burn rate is 14.41.
| Alert | Burn rate | Long window | Budget consumed | Days to exhaust if it continues |
|---|---|---|---|---|
| Page | 14.4 | 1 hour | 2% | 2.08 |
| Page | 6 | 6 hours | 5% | 5.00 |
| Ticket | 1 | 3 days | 10% | 30.00 |
Each row is the burn rate times the window divided by 720 hours; the percentages match the workbook's table and the exhaustion times follow by dividing 30 days by the burn rate1. The pattern to say aloud is that fast burns page a human and slow burns open a ticket. Two more numbers help in any answer: a request that needs three dependencies, each 99.9 percent available, can be no more than 99.9 percent available and, if their failures are independent, is about 99.7 percent available (0.999 multiplied three times); with five it is about 99.5 percent1. So a service cannot promise more than its critical dependencies allow.
Questions interviewers ask on SRE
| Question (our wording) | What a strong answer says |
|---|---|
| How much downtime does a 99.99 percent target allow? | 4.32 minutes in 30 days, or 52.56 a year; each extra nine cuts it tenfold. |
| What would you page on, and what would you only graph? | Page on symptoms users feel, such as errors and latency against the objective; graph causes unless they are definite and imminent. |
| An alert fires and users report slowness; what do you do first? | Confirm impact and scope, check recent changes, mitigate by rolling back or shifting traffic, communicate, then find the root cause. |
| How would you size capacity for a launch? | Estimate the peak, add headroom, and check the service still meets its objective if a quarter of capacity fails. |
| How do you release a risky change? | Canary it to a small share of traffic, watch the symptom alerts, roll back fast, and release only while budget remains. |
| Tell me about toil you removed | Name the manual task, how often it ran, what you automated and the time it gave back, with a number. |
The interview loops guide shows where an SRE round sits in a loop.
Follow-up questions
How do I prepare for a site reliability engineer interview?
Learn the six areas in order, do the downtime and burn-rate arithmetic until it is automatic, and prepare two incident stories and one toil story with numbers. Practise the incident scenario out loud, because the order of your moves is what is judged.
What is the difference between SRE and DevOps?
The Google book says one could view SRE as "a specific implementation of DevOps with some idiosyncratic extensions". In an interview, say they share goals, and that SRE adds explicit reliability targets, error budgets and a cap on operational work.
What is asked in an SRE interview?
Expect a systems or coding round, a troubleshooting or incident scenario, a design round with reliability in mind, and a behavioural round on incidents. Troubleshooting rounds lean on networking fundamentals, such as name resolution and connection set-up.
What is an error budget and what happens when it runs out?
An error budget is the unreliability a service may spend, equal to one minus its objective over a period. When it is used up, the book describes temporarily halting releases while more is invested in testing and development; subtler policies slow or roll back releases as it nears exhaustion.
Why not aim for 100 percent availability?
The SRE book says 100 percent is probably never the right target, since it is impossible to achieve, more reliability than users notice, and it slows feature releases. The questions to ask in return are on the questions to ask the interviewer page.
- An SLI is what you measure, an SLO is the target, and the error budget is what the target leaves to spend.
- Each extra nine cuts the allowed downtime tenfold: 43.2, 4.32, then 0.432 minutes in 30 days.
- Page on symptoms; use the four golden signals: latency, traffic, errors, saturation.
- In an incident, mitigate first and find the root cause second.
- Fast burn rates page a human, slow ones open a ticket, and toil stays under half of the job.
Sources
- Introduction, Site Reliability Engineering (Beyer, Jones, Petoff, Murphy, eds.) — Google, O'Reilly Media (page notice: copyright 2017, CC BY-NC-ND 4.0), 1 Jan 2017
- Service Level Objectives, Site Reliability Engineering — Google, O'Reilly Media (page notice: copyright 2017, CC BY-NC-ND 4.0), 1 Jan 2017
- Embracing Risk, Site Reliability Engineering — Google, O'Reilly Media (copyright 2017), 1 Jan 2017
- Monitoring Distributed Systems, Site Reliability Engineering — Google, O'Reilly Media (copyright 2017), 1 Jan 2017
- Eliminating Toil, Site Reliability Engineering — Google, O'Reilly Media (copyright 2017), 1 Jan 2017
- Postmortem Culture: Learning from Failure, Site Reliability Engineering — Google, O'Reilly Media (copyright 2017), 1 Jan 2017
- Alerting on SLOs, The Site Reliability Workbook — Google, O'Reilly Media (page notice: copyright 2018, CC BY-NC-ND 4.0), 1 Jan 2018
