In this article
Before you start
This page belongs to the machine learning system design interview learning path, and it assumes only three ideas.
Why interviewers ask this
A rollout question separates candidates who treat a launch as one number from candidates who treat it as a decision under risk. Candidates report it in machine learning system design rounds, often once the model is designed and the interviewer asks how you would know it is safe to release.
A weak answer names an accuracy score or a click rate. A strong answer gives each metric a role, says which checks come first, and says what blocks the launch. A Staff-level answer adds three things: the margins are agreed before the data arrives, the test has enough users to see a regression, and the model is released in stages. The rest of this page builds that answer from zero, so you can say it in about a minute. The question also tests whether you will say no to a launch that looks good on the headline metric.
Rollout metrics from first principles
- Non-inferiority margin
- A non-inferiority margin is the largest worsening of a guardrail metric the team agrees to tolerate, fixed before the test, so the guardrail passes only when the whole confidence interval of the change stays inside that margin.
In plain words
Think of a go or no-go call before a flight. The headline question is whether the plane is faster. The crew first checks the instruments. Then they check fuel, brakes and weather. A faster plane with a failing brake does not take off. The analogy breaks in one place. A launch metric is noisy, so each check is a statistical test with a margin for error.
In one precise sentence, a rollout decision is a rule fixed in advance. Roll out when the trust checks pass, the success metric is better, and every guardrail stays inside its margin. The margin is fixed in advance so that nobody can move it after seeing the numbers to rescue a launch they already like.
Step 1: Check the experiment is valid before reading any metric
A sample ratio mismatch check runs before any metric is read, because a broken split makes every metric untrustworthy. Before the test, the team configures the split between treatment and control, typically one to one, and during the test it compares the observed counts with that split using a chi-squared test. Microsoft states that every A/B test must first pass this check before its effects are analyzed, and that results with a mismatch should not be trusted until the root cause is diagnosed.
Take a hypothetical 50/50 test. Counts of 50,000 and 48,800 give a chi-squared statistic of 14.57 and a p-value of about 0.00013, while 50,000 and 49,800 give 0.40 and about 0.53 (illustrative counts).1 The sources quote no p-value cutoff, so the threshold is a team choice. In one Microsoft case, users on version B were so engaged that a bot-detection filter removed them, which raised the warning.
Step 2: Guardrails veto the success metric
A guardrail breach blocks a rollout even when the success metric is up. The success metric is the measure the change exists to move, and ideally the team picks one. An overall evaluation criterion (OEC) is a quantitative measure of an experiment's objective that can be measured within the test and is believed to drive long-term goals (Kohavi, Tang and Xu, 2020). Revenue alone is insufficient, so Bing weighs revenue against user-experience metrics such as sessions per user.
Guardrails cover what the change must not touch; the book's example is software crashes, and it says safety can be handled by a heavy weight in the OEC or, equivalently, as a guardrail. Spotify's platform writes the rule down: a comparison ships if at least one success metric is significant, or none is required, and all guardrail non-inferiority tests pass (Schultzberg, 2026). A pure rollout checks guardrails only. When the outcome arrives slowly, as in churn and credit-risk prediction, the success metric must be a short-term stand-in.
Step 3: Fix margins and sample size, then read the interval
Margins and sample size are fixed before the test starts. A non-inferiority margin is a business decision, and the sources give no universal value, so every margin on this page is hypothetical. The test must also be able to see a regression if one exists. For a two-sample z-test the users per variant are n = 2σ²(z_α/2 + z_β)² / δ², where δ is the smallest effect worth detecting (Schultzberg, 2026), and Microsoft recommends at least 80% power for success and guardrail metrics. At 80% power and a two-sided 0.05 level, n is about 15.7σ²/δ².1
Then read the interval, not only the verdict. Schultzberg gives two results that are both +4.5%, one with an interval of [+4.1%, +4.9%] and one with [+0.1%, +9.0%]; the second is much less trustworthy (Schultzberg, 2026).
Step-by-step description
- The split must match the configuration.
- A guardrail breach vetoes the success metric.
- Margins and power are fixed before the test.
- A sample ratio mismatch check runs before any metric is read.
- A guardrail breach blocks a rollout even when the success metric is up.
- Margins and sample size are fixed before the test starts.
Three illustrative results, decided
Three hypothetical results show the rule at work. Sessions per user is the success metric, new-user retention is the guardrail, and the margin is minus 1.0% relative; each interval is the estimate plus or minus 1.96 standard errors (illustrative figures).1 Roll out when the success interval is above zero and the guardrail's lower end is above the margin.
| Case | Success: sessions per user | Guardrail: new-user retention | Verdict |
|---|---|---|---|
| A | +1.2% [0.42%, 1.98%] | -0.1% [-0.69%, 0.49%] | Roll out |
| B | +2.0% [1.02%, 2.98%] | -0.8% [-1.39%, -0.21%] | Block: lower end below -1.0% |
| C | +0.2% [-0.78%, 1.18%] | 0.0% [-0.59%, 0.59%] | No lift: ship only a pure swap |
Case B has the largest lift but fails its guardrail margin, so it is blocked. Case C shows no significant lift, which is a normal outcome: only one third of ideas tested at Microsoft improved the metrics they were designed to improve, and roughly 70% of tested effects are true nulls (Berman and Van den Bulte, 2022, cited by Schultzberg). A cheaper model replacing the current one is a pure rollout, so Case C ships if the guardrails pass. The test type, A/B test or pure rollout, is fixed before the test starts; relabeling a flat A/B test as a rollout after the data arrives would be moving the goalposts.
How many users the test needs
Sample size shows why power matters. Take the formula from Step 3, n = 15.7σ²/δ² at 80% power. For a yes-or-no metric with a 10% baseline rate, the variance σ² is 0.10 × 0.90 = 0.09 (illustrative baseline). If the smallest effect worth detecting, δ, is 0.5 points, which is 0.005 as a rate, each variant needs a sample of about 56,512.1 Because δ is squared in the denominator, halving the smallest effect you want to detect quadruples the sample needed per variant: 3,532 per variant for 2 points, 14,128 for 1 point and 56,512 for 0.5 points.1 A guardrail test with too few users cannot see a regression, so this arithmetic belongs before the launch, not after.
Text description of the diagram
- Halving the smallest effect you want to detect quadruples the sample needed per variant.
- At 80% power, a 10% baseline rate and a 0.5-point effect, each variant needs a sample of about 56,512.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What metric are we using here to decide whether we can roll out or not?
I would not use one number. I would write the decision rule before the test starts. First, a trust check: the traffic split must match what we configured, otherwise I do not read any metric. Second, one primary success metric tied to the product goal, such as sessions per user for a recommender, rather than revenue alone.
Third, guardrails with agreed margins, such as crashes, latency, complaints and any segment we must not hurt; if one breaches, we do not roll out even when the primary is up. I check the confidence interval, not only significance, and I size the test so guardrails have enough power. For a model, I would stage it: shadow, small canary, then ramp, with a rollback trigger on the guardrails.
What makes it Staff-level: it names the order of the checks, the veto, and the staged release, and it fixes the margins before the data arrives.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Metric choice | Names accuracy or click rate. | Names one success metric tied to the goal. | Picks one primary metric, adds a short-term stand-in for slow outcomes, rejects revenue alone. |
| Safety | Ignores what could get worse. | Lists guardrails such as latency and crashes. | Fixes margins before the test; a guardrail breach vetoes the launch. |
| Validity | Reads results straight away. | Mentions checking the traffic split. | Runs the split check first and diagnoses a mismatch before reading anything. |
| Statistics | Ships on any significant p-value. | Reads the interval and the effect size. | Sizes the test so guardrails have enough power to catch a regression. |
| Release path | Flips the change for everyone. | Ramps traffic in steps. | Runs shadow, canary and ramp, with a rollback trigger on the guardrails. |
Three variants of this question
What metric would you use to decide whether a new ranking model can launch?
Require the new model to beat the current one on a held-out set by more than run-to-run noise, because a win there can still lose in production through skew, latency or effects on users. A shadow deployment is a release where the new model's outputs are logged but never shown, and a canary release is one to a small share of live traffic watched against the guardrails. Ramp only if both pass; the retrieval, ranking and re-ranking funnel shows which stage the model sits in.
How do you decide whether a cheaper model can replace the current one?
A cheaper model replacing the current one is a pure rollout, so no success metric must improve: the decision is whether every guardrail passes its non-inferiority test (Schultzberg, 2026). Set the margin for quality and user-experience guardrails first, then check that cost or latency actually fell.
What if the main metric is flat but the team wants to ship?
A flat result is common, but flat does not prove no effect if the metric was underpowered. Check the interval and the power; if the change is a deliberate swap and every guardrail passes, ship it, and otherwise do not ship on hope.
Common mistakes
- "We ship because the primary metric is up." (wrong) A guardrail breach vetoes the launch. (right) "We ship if the success metric is up and every guardrail stays inside its margin."
- "The p-value is below the threshold, so the lift is real and big." (wrong) Significance says nothing about size. (right) "I report the effect and its interval, because the same estimate can come with a precise or a very wide interval."
- "The guardrail looks flat, so nothing broke." (wrong) A guardrail without enough users hides regressions. (right) "I size guardrails for at least 80% power, as Microsoft recommends."
- "A better score on the held-out set means roll out." (wrong) It only earns the next stage. (right) "A held-out win earns a shadow run and a canary, not a full release."
Follow-up questions
Is a statistically significant lift enough to roll out?
No. Significance says the effect is probably not zero; the interval says how big it may be, and the guardrails and the split check still apply (Schultzberg, 2026).
Do more guardrail metrics raise the false-positive rate?
No. Shipping requires every guardrail to pass, so adding guardrails cannot raise the chance of a false ship; it raises the chance of a false block and costs power. The correction belongs to the success metrics: with independent metrics, shipping when any one of eight is significant at 0.05 carries about a 34% chance of a false positive (computed as 33.7%).1 With two success metrics and ten guardrails, Spotify's approach corrects only over the success metrics, 0.05/2 = 0.025, rather than 0.05/12 ≈ 0.004 (Schultzberg, 2026).
What do you do if the sample ratio check fails?
Stop reading the metrics and find the cause, because a mismatch is a symptom of quality issues such as a filter that removes users from one variant. Re-run only after the fix.
How many guardrail metrics should a rollout have?
No standard exists. In a sample of Spotify experiments, the median was four guardrail metrics and two success metrics per experiment, with means of 5.7 guardrail metrics and 3.7 success metrics (Schultzberg, 2026). Choose the few things you would refuse to degrade, each with a margin and enough power.
Can you use a single metric to decide a rollout?
Ideally a product has just one OEC metric, but other metrics are still needed to make sure the decision is correct. The single metric can be a weighted combination.
What guardrails fit a fraud or pricing model?
Fraud and pricing models change other people's behavior, so guardrails include review queue length for real-time fraud detection and refund requests for dynamic surge pricing. A moderation model needs user-experience guardrails too, as in harmful-content detection.
- A rollout metric is a decision rule: trust checks, a success metric and guardrails.
- Check that the traffic split matches the configuration before reading any metric.
- A guardrail breach blocks the launch even when the success metric is up.
- Fix margins and sample size before the test, and read the interval, not only significance.
- Stage a model release: shadow, canary, ramp, with a rollback trigger on the guardrails.
Sources
- Trustworthy Online Controlled Experiments, front matter and Chapter 1 (Kohavi, Tang, Xu) — Cambridge University Press (2020), companion site; accessed 2026-10-26, 1 Jan 2020
- Nobody Puts Bonferroni in a Corner (arXiv 2604.09256, v1, preprint) — arXiv (Schultzberg, Spotify); accessed 2026-10-26, 10 Apr 2026
- Diagnosing Sample Ratio Mismatch in A/B Testing (page year 2020) — Microsoft Research, Experimentation Platform; accessed 2026-10-26, 1 Jan 2020
- Alerting in Microsoft's Experimentation Platform (page undated; text refers to a 2020 incident) — Microsoft Research; accessed 2026-10-26, 1 Jan 2020
- A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls (summary of Dmitriev et al., KDD 2017) — the morning paper (Adrian Colyer); accessed 2026-10-26, 25 Sep 2017
