How do you build an on-call culture as a manager?

In this article

1. Before you start
2. Why interviewers ask this
3. On-call culture from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the site reliability engineering (SRE) path, covered in the site reliability engineer interview questions guide. The wider background is in the DevOps interview questions path.

Why interviewers ask this

A manager who gets on-call wrong loses people and reliability together. Tired engineers leave, and the ones who stay stop trusting the pager. So the prompt "how would you build a healthy on-call culture?" tests whether you see on-call as a system you tune, not a rota you publish.

Google's reliability book says that SRE managers are responsible for keeping on-call sustainable, which gives you the frame: the duty has a size and a quality, and both can be managed.

A weak answer lists a rota and a stipend. A strong answer names two measures, a time budget, a rule for what may page, the support people need and a learning loop. A Staff-level answer also says what you do when the service stays too noisy: the owning team takes the fix before more people take the pager. The live-incident side of the same topic is in how to answer an incident response scenario interview.

On-call culture from first principles

On-call

On-call is a duty in which an engineer is available for a set period and ready to respond to production incidents during that time with appropriate urgency, as the Google SRE Workbook on-call chapter describes it. It usually implies some work outside normal hours.

Blameless postmortem

A blameless postmortem is a written review of an incident that looks for causes in systems and processes instead of in individual people, and ends with tracked actions. Its value is proportional to the learning it creates, per the Google SRE Workbook postmortem chapter.

In plain words

Think of a fire station. Someone is always ready. There is a clear order for who goes first and who backs them up. Every call is reviewed afterwards so the next one goes better. The analogy breaks in one place. Firefighters wait for rare emergencies, but an on-call engineer can be flooded by noise that their own team created and can fix.

In one precise sentence, an on-call culture is the set of habits, limits and shared expectations that decide how the duty is measured, shared, supported and learned from.

The five steps below follow that order. A manager answers the interview question by walking through them.

Step 1: Measure quantity and quality, not only hours

Quantity is the share of time an engineer spends on call, and quality is the number of incidents they handle per shift. The Google SRE book measures balance on exactly these two axes. Most candidates talk only about the first.

The book also sets a time budget: at least 50% of SRE time goes into engineering, and of the remainder no more than 25% can be spent on-call, leaving up to another 25% for other operational work.

Quality needs a ceiling. The workbook targets a maximum of two incidents per on-call shift, so there is time for follow-up. The book's own reckoning is that one incident takes six hours, which gives two incidents per 12-hour shift.¹

You now know what to track: percent of time on call and incidents per shift. The next step uses the first number to size the team.

Step 2: Size the rotation so the budget holds

A rotation must be big enough that nobody exceeds the on-call cap; the capacity planning and on-call questions guide covers the sizing side in more depth. The Google SRE book gives the minimum for a single-site team as eight engineers, assuming week-long shifts with a primary and a secondary: each engineer is on call for one week every month. For dual-site teams a reasonable minimum is six per team.

The arithmetic is simple. Two slots a week shared across n engineers puts each person on call 2 divided by n of the weeks. In a rotation of three, each engineer is on call 66.7% of weeks. In a rotation of five it is 40%. In a rotation of eight it is 25%, exactly the cap.¹

Too little on-call is also a risk. The book says teams should be sized so that every engineer is on call at least once or twice a quarter, because skill and confidence fade otherwise.

Step 3: Page only on actionable alerts

A page is justified only when a human must take action at once. The SRE book says all paging alerts should be actionable, and that noisy alerts producing more than one alert per incident should be tuned toward a 1:1 alert-to-incident ratio.

How fast a person must answer comes from the target. A service level objective (SLO) is a target for how well a service must work, such as an availability percentage. The book gives typical response times of 5 minutes for user-facing or highly time-critical services and 30 minutes for less time-sensitive ones. At 99.99% quarterly availability the allowed downtime is around 13 minutes, so minutes matter.¹ At 99.98% the workbook example is roughly 26 minutes per quarter.

Priorities keep the pager quiet. The workbook describes an Evernote scheme: the top priority is immediately actionable, SLO-impacting and pages on-call; the middle priority waits for the next business day as an email or ticket; the lowest is informational and lives on dashboards. Low-priority alerts that page cause fatigue.

Step 4: Give responders escalation paths, playbooks and handoffs

People handle a bad night better when the system carries some of the load. The SRE book names the most important on-call resources: clear escalation paths, well-defined incident-management procedures and a blameless postmortem culture.

A playbook is a written entry, created whenever an alert is created, that tells the responder what the alert means and what to try first. The workbook says playbooks reduce stress, the mean time to repair (MTTR) and the risk of human error. If a playbook is a fixed list of commands run every time an alert fires, automate it.

Handoffs keep context: read the previous shift's handoff when you start, and send one when you finish. Training matters too. The workbook's Google example has new team members shadow first, then take primary on-call with the old team as backup, and use "Wheel of Misfortune" role-play; new teams are not expected on call for three to nine months. Psychological safety, it says, is vital for effective on-call rotations.

Step 5: Learn from each incident with blameless postmortems

The loop closes when incidents change the system. The postmortem culture chapter says a truly blameless postmortem culture results in more reliable systems. The interview version of this step is in how to give a blameless postmortem interview answer. Finger-pointing makes people risk-averse because they fear public shaming, so they hide what happened.

Blameless does not mean no accountability. Postmortem actions need an owner and tracking, and vague items such as "Improve" or "Make better" are listed as signs of a bad postmortem, along with missing context and late publication. In the words of Ben Treynor Sloss, quoted in the same chapter: "To our users, a postmortem without subsequent action is indistinguishable from no postmortem."

The chapter also argues that changing automated systems and processes is more reliable than changing human behaviour. So when the same page keeps firing, the manager's lever is a fix or automation, not a lecture.

1 · Measure2 · Size3 · Page4 · Support5 · LearnReplay

Step-by-step description

1. Track time on call and incidents per shift.
2. Size the rotation to the budget.
3. Page only on actionable alerts.
4. Give escalation paths and playbooks.
5. Run blameless postmortems with owned actions.

• The five steps are measure, size, page, support and learn.
• Postmortem actions need an owner and tracking.

The five steps are measure, size, page, support and learn.

Worked example

Suppose a team of five covers a user-facing service with week-long shifts and a primary and a secondary. Each engineer spends 40.0% of their weeks on call, above the 25% cap, so the team needs eight people or a second site before it can honour the budget.¹

Now the pager. Suppose, as an illustration, that a noisy 24-hour shift fires 100 pages from 10 incidents, so each incident fires ten pages: a ratio of 10 to 1 against the 1 to 1 target. After grouping, 10 pages remain.¹

Grouping fixes noise, not volume. Ten incidents at six hours each need 60 hours of work, and a 24-hour shift holds at most four such incidents, while the workbook target is two or fewer per 12-hour shift.¹ The manager's job is then to remove causes, not to add more people to absorb them.

In a rotation of three, each engineer is on call 66.7% of weeks. A rotation of eight puts each engineer on call 25% of weeks.¹

Text description of the diagram

• In a rotation of three, each engineer is on call 66.7% of weeks.
• A rotation of eight puts each engineer on call 25% of weeks.

Ten incidents at six hours each need 60 hours. A 24 hour shift holds at most four such incidents.¹

Text description of the diagram

• Ten incidents at six hours each need 60 hours.
• A 24 hour shift holds at most four such incidents.

Trade-offs and edge cases

On-call is not only for large reliability teams. The workbook says it is not exclusively an SRE function: many developer teams are directly on call for their service. Ownership by the team that builds the service puts the pain next to the fix. The monitoring and alerting basics matter for every team that owns a pager.

More pages do not mean a more responsive team. The target is few incidents per shift, so a high count means noise or fragility, and a manager should read it as a defect rate. A quiet pager is not purely good either, because the book warns that too little on-call erodes skill and confidence.

If a development team's service is too noisy, the SRE book notes that in extreme cases SRE teams may have the option to "give back the pager". That hands the duty to the owners until the service meets the bar.

Two things this page does not settle. Night shifts have health costs that the book cites from outside research, and follow-the-sun designs depend on team layout; the book's dual-site minimum of six per site is the figure to quote. Compensation norms beyond Google's described approach vary by company, so say that practice differs and name the principle: out-of-hours work should be compensated. The wider reliability toolset is covered in the rest of the SRE path.

How to answer it in an interview

The spoken structure: measure two axes, size the rotation, page only on actionable alerts, support responders, then close the loop with postmortems.

Common mistakes

• "We have a rota and a stipend, so on-call is handled." (wrong) That covers quantity only. (right) "I also track incidents per shift and keep it at two or fewer."
• "Every alert should page so nothing is missed." (wrong) It causes fatigue. (right) "A page must be actionable, and lower-priority alerts become tickets or dashboard items."
• "More pages show the team is responsive." (wrong) It shows load. (right) "A high page count tells me to fix the causes."
• "Blameless means nobody is accountable." (wrong) Actions then never get done. (right) "People are not blamed, but every action has an owner and a tracking item."
• "Recurring pages mean we need to train people harder." (wrong) The system stays the same. (right) "I change the automation or process, and I automate fixed playbook steps."

Follow-up questions

What do you do when the developer team's service keeps paging SRE?

Set the target at one alert per incident and at most two incidents per shift, and hold regular production meetings with the developer team to go through the noisy alerts. If the service still does not meet the bar, the SRE team may have the option to "give back the pager", so the owners carry it until it does.

How many engineers do you need for an on-call rotation?

The SRE book gives eight as the minimum for a single-site team on week-long shifts with a primary and a secondary, and six per team for dual-site teams. Smaller teams need tighter alert hygiene and may use shorter shifts or shared ownership.

How do you reduce alert fatigue on a noisy rotation?

Remove alerts that are not actionable, group duplicate alerts toward one alert per incident, and move low-priority signals to tickets or dashboards. Then automate any fixed playbook steps. The monitoring vs logging vs tracing page covers the monitoring basics behind this.

Should engineers be paid for being on call?

The workbook says on-call usually implies out-of-hours work and that this work should be compensated. The book describes time off in lieu or cash, capped at some proportion of salary. Say that practice varies and that the principle is fair compensation.

Does a small team need on-call practices from big companies?

Yes, adapted. The workbook chapter replies to the complaint "We are not Google; we're much smaller" by showing tiered alert priority, playbooks, handoffs and paging based on the service level objective, all of which fit a small team.

What does a blameless postmortem change?

It makes people share what happened, so the team can fix systems instead of hiding errors. It only works with owned, tracked actions. For the live-incident side, read how to answer an incident response scenario interview.

Sources

1. Being On-Call (Chapter 11, Site Reliability Engineering) — Google SRE, 1 Apr 2016
2. On-Call (Chapter 8, The Site Reliability Workbook) — Google SRE, 1 Jul 2018
3. Postmortem Culture: Learning from Failure (Chapter 10, The Site Reliability Workbook) — Google SRE, 1 Jul 2018