Pre-deployment metrics looked great, the A/B test did not: why?

In this article

1. Before you start
2. Why interviewers ask this
3. Pre-deployment metrics versus A/B tests from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the machine learning (ML) system design interview guide, which shows where evaluation fits in a full design round.

Why interviewers ask this

A model that passes its checks and then fails in production costs a team weeks, so interviewers want to know whether you can explain why the two numbers disagree. Candidates report questions such as "you trained the model; is it ready to roll out?" and "how would you evaluate the model once it is deployed?" in ML system design rounds.

A weak answer says "the A/B test must be noisy" or "we need more data." A strong answer names the layer where the gap opened: the metric is a proxy, the logged data is biased, the serving path differs from training, or the test is under-powered. It also says which check would have caught each one before launch. A Staff-level answer adds a rollout plan that treats the pre-deployment metric as a filter, not as proof.

Pre-deployment metrics versus A/B tests from first principles

Pre-deployment metric

A pre-deployment metric is a score computed on held-out or logged data before a model meets live users, such as accuracy or ranking quality. It filters out broken models cheaply, but it measures the data you already have, not the behavior of users who will see the new model.

Proxy metric

A proxy metric is an easy-to-measure number used in place of the goal you actually care about, such as click rate standing in for user satisfaction. A model can improve the proxy without improving the goal.

In plain words

Think of a driving test on a closed course. A driver can ace it and still struggle in city traffic. The course tests something real, but it is not the road.

A pre-deployment metric is the closed course. The A/B test is the road. The analogy breaks in one place: the course was built by an examiner, but your test data was produced by the old model. So the course itself leans toward the old model's habits.

In one precise sentence, a pre-deployment metric and an A/B test disagree when they measure different things, on different data, under different conditions, with different amounts of noise.

Step 1: The metric is a proxy for the goal

A model is trained to improve a number, and that number is rarely the goal. Zinkevich's Rules of Machine Learning says the objective should be easy to measure and a proxy for the "true" objective, and notes that teams often do not know what the true objective is and often disagree about it.

Input: a goal such as "users find what they need." Output: a measurable stand-in such as click rate or ranking quality on labeled queries. The stand-in leaves things out. A click can mean interest or a misleading title; a labeled query set can miss the queries that matter next month.

The same guide says that if you push up the directly optimized metric but decide not to launch, the objective may need revision, and that indirect effects make good metrics for A/B tests and launch decisions. You now know the first gap: the metric can rise while the goal stands still.

The number itself can also be wrong. Leakage is when a feature or label carries information the model would not have at request time, and it inflates the held-out score. A random split can leak the future into training, and tuning again and again against one held-out set overfits it. Split by time instead: the same guide advises testing on data gathered after the training data, so a model trained up to January 5 is tested on data from January 6 onward.

Step 2: Logged data only holds what the old model showed

Logged data is biased toward the old model's choices, which makes a pre-deployment score on it unreliable for a new model. The Rules of Machine Learning guide warns about a common pattern: logging only what the user saw. All the negative examples for unseen items are then dropped, so the data cannot say why an item was not seen.

Input: a log of items the old model displayed, with clicks. Output: labels for those items only. A new model that ranks a different item first has no label for it. A naive evaluation counts a missing label as "no click," which punishes any model that disagrees with the old one and rewards copies of it.

This gap works in both directions. A model can look worse than it is, as the worked example shows, or look better, when it is tuned to the quirks of the log. Common remedies are to log a small random slice of results, or to use propensity-weighted (counterfactual) estimators that correct for items the old model rarely showed (Joachims, Swaminathan and Schnabel, 2016). The search ranking design page and the retrieval, ranking and re-ranking funnel both depend on logs of this kind.

Step 3: Training and serving can disagree

Training-serving skew is a difference between how the model is fed during training and how it is fed in production, and it can lower live quality while the pre-deployment score stays high. The Rules of Machine Learning guide says the model in the training environment should give the same score as the model in the serving environment.

Input: the same user request, run through the training pipeline and through the production pipeline. Output: two scores that should match. They differ when a feature is computed by different code, when a lookup table is stale, or when a feature that exists in the training log is missing at request time.

The guide describes a case where a joined table stopped updating and quality decayed gradually; refreshing a table that had been stale for six months lifted the install rate of one product by two percent. Nothing crashed, so no alert fired. Check for skew by logging the features at serving time and comparing them with the training values.

Step 4: The A/B test measures real users, with noise

An A/B test measures a different quantity from a pre-deployment metric: real user behavior, over days, with random variation that can hide a real effect. Four things go wrong.

First, the test lacks statistical power, which is the chance that a test detects a real effect of a given size. A small true lift needs a large sample. Second, the assignment is broken: a sample ratio mismatch means the group sizes differ from the plan by more than chance allows, which points to a bug in assignment or logging (Fabijan et al., 2019).

Third, a novelty effect is a temporary change in behavior because something is new, so a short test may not show the settled effect; plot the lift by day. Fourth, interference is spillover between groups, for example through shared inventory, a marketplace or a social network, so the control group is affected by the treatment; running longer does not fix it, but randomizing by cluster or region can.

Input: two randomly assigned groups. Output: a difference with an uncertainty range. A flat result with a wide range means "unknown," not "the model failed." The worked example puts numbers on this.

1 · Proxy metric2 · Logged data3 · Live serving4 · A/B testReplay

Step-by-step description

1. The metric stands in for the goal.
2. Labels exist only for shown items.
3. Features can differ from training.
4. Real users, with noise.

• A held-out metric measures a proxy for the goal.
• Logged data holds only what the old model showed.
• Live serving can compute features differently from training.
• An A/B test measures real users with noise.

Worked example

This example is a simulation with illustrative inputs, computed by a short script.¹ It has two parts: the logged-data trap, then an A/B test that looks like a failure and may not be.

Part 1, the logged-data trap. Take 20,000 queries with 10 candidate items each, and give every item a hidden true click probability (illustrative). The old ranker scores items with that probability plus noise, and shows its top three. Only those three get labels. Three models are then scored by "clicks on the logged items in their own top three," with unlabeled items counted as zero. The "near copy" is the old ranker plus a little extra noise. The "better ranker" uses the true probability with small noise.¹

The better ranker picks items the old model never showed, so most of its choices carry no label and count as zero. A team that trusted this score would reject the best model. The reverse mistake also exists: a model tuned to the quirks of the log can score high and gain nothing live.

Text description of the diagram

• The better ranker scores 0.6263 on logged data but earns 0.9306 true expected clicks per query.
• The old ranker scores 0.7994 on both measures.

Part 2, a flat A/B test. Suppose a new model had a better pre-deployment score, and the test puts 100,000 visitors in each arm (illustrative). Control records 5,000 conversions, a rate of 5.00%; treatment records 5,120, a rate of 5.12%. The lift is 0.12 percentage points, or 2.4% relative. The two-proportion z statistic is 1.224 and the p-value is 0.2209, so the test cannot rule out chance. The 95% confidence interval for the difference runs from -0.072 to +0.312 percentage points.¹

The interval includes both "slightly harmful" and "clearly useful." The test is inconclusive. The question is how many users it needed. With a baseline of 5% and the usual 5% significance level and 80% power, the normal-approximation formula is:

Here $n$ is users per arm, $p$ the baseline rate (used for both arms), $\delta$ the lift to detect, $z_{1-\alpha /2}$ is 1.96, the two-sided critical value for a 5% significance level, and $z_{1-\beta }$ is 0.8416 for 80% power.¹

Text description of the diagram

• Detecting a lift of 0.5 percentage points needs 29,826 visitors per arm.
• Detecting a lift of 0.1 percentage points needs 745,644 visitors per arm.

Last, check the split before reading any result. A planned 50/50 test that lands on 100,000 and 101,500 visitors has a z statistic of -3.342 and a p-value of 0.0008, so the split is broken (a sample ratio mismatch, the kind of failure Fabijan et al., 2019 diagnose) and the lift cannot be trusted.¹

Trade-offs and edge cases

Pre-deployment metrics are cheap and fast, and A/B tests are slow and costly, so the two work best as stages, not rivals. The Rules of Machine Learning guide still recommends a sanity check on held-out data right before export, so the pre-deployment score is a gate that catches broken models. It is not proof of product impact.

Edge cases: a label may arrive long after the event, as with confirmed fraud (see fraud detection design), so the pre-deployment score uses old or partial labels. Rare-event problems such as anomaly detection have so few positives that both the metric and the test are noisy. In harmful-content detection, the cost of a miss is not equal to the cost of a false alarm, so a single score hides the trade-off.

Common middle steps between the two stages include shadow runs (the new model scores live traffic without affecting users, so they check serving behavior and score drift, not user effect), a small canary release (it checks for breakage and slowness on a small share of live traffic), and interleaving, which mixes results from two rankers in one list so that users' clicks compare them directly.

How to answer it in an interview

The spoken structure: say the gap has layers, name the main causes, say how you would check each, then say how you would change the rollout.

Common mistakes

• "A better pre-deployment score means a better product." (wrong) The score measures a proxy on logged data. (right) "The score tells me the model is not broken; the A/B test tells me whether users benefit."
• "The A/B test is noisy, so I ignore it." (wrong) Noise is a reason to read the interval and the power, not to discard the result. (right) "A flat result with a wide interval means the test was too small to say."
• "No lift means the model failed." (wrong) The interval may include a useful effect. (right) "No significant lift means unknown; I check the sample size against the effect I care about."
• "I trust the lift without checking the split." (wrong) A broken split invalidates the comparison. (right) "I run the sample-ratio check before I read any result."
• "Pre-deployment metrics are useless." (wrong) They are a cheap filter that blocks broken models. (right) "I use them as a gate, not as the launch decision."

Follow-up questions

How do you get the ground truth for a pre-deployment metric?

Ground truth comes from three places: user behavior in logs (clicks, purchases, skips), human raters who label a sample, and delayed outcomes such as confirmed fraud. Each has bias: logs only cover shown items, raters are costly and subjective, and delayed labels arrive late. State which one you use and what it leaves out.

How would you evaluate the effectiveness of your ML models under deployment?

Track the live metric you care about, compare it with a control group where possible, and monitor input features for drift and missing values. Add alerts for silent failures such as a stale table. Retrain on a schedule and re-run the pre-deployment check each time.

You trained the model; is it ready to roll out?

Not yet. The held-out score shows it is not broken; it does not show product impact. Confirm that serving scores match training scores, then roll out in stages (shadow, canary, A/B test) and set the success metric and sample size before the test starts. See the ML system design guide for where this fits.

Why did the A/B test show no lift when the pre-deployment metric improved?

Possible reasons are a proxy mismatch, biased logged data, skew between training and serving, or a test too small to detect the lift. Check the confidence interval first: if it is wide and includes zero, the result is inconclusive, not negative.

What is a sample ratio mismatch?

A sample ratio mismatch is a group split that differs from the plan by more than chance allows, such as 100,000 against 101,500 visitors in a planned 50/50 test. It points to a bug in assignment or logging, so you fix it and rerun before trusting any lift.

Sources

1. Rules of Machine Learning: Best Practices for ML Engineering (accessed 2026-10-26; page undated) — Google for Developers (Zinkevich), 26 Oct 2026
2. Unbiased Learning-to-Rank with Biased Feedback (arXiv 1608.04468) — arXiv (Joachims, Swaminathan, Schnabel), 16 Aug 2016
3. Diagnosing Sample Ratio Mismatch in Online Controlled Experiments — ACM KDD 2019 (Fabijan et al.), 4 Aug 2019