In this article
Before you start
This page is one answer in the large language model (LLM) evaluation guide. The causes below apply to classic models and to large language model (LLM) applications alike.
Why interviewers ask this
A model that scores well before launch and badly after it costs the team a release and the trust of whoever approved it. Interviewers ask this question, reported in interviews at AI labs and large tech companies, to see whether you treat a held-out score as an estimate that can fail, not as a fact about the model.
A weak answer says "the model overfit" and stops. A strong answer separates four causes (leakage, pipeline mismatch, a change in the data or traffic, and a metric that is not the goal) and gives one check for each. A Staff-level answer orders the checks by cost, says how to monitor so the gap does not return, and states that only an online experiment settles whether a change helps users.
The held-out-versus-online gap from first principles
- Held-out score
- A held-out score is a metric that is measured on examples the model never trained on; it estimates performance on similar data but can differ from online results when the inputs, the sample or the metric change.
In plain words
Think of a driving test taken on an empty car park. You pass with a high score. Then you drive in rush hour and do worse. The test was not wrong about the car park. It measured something slightly different from the road.
A held-out score works the same way. It is a measurement on one sample, with one set of inputs, using one metric. Online results come from a different sample, different inputs and often a different metric. The analogy breaks in one place: a driving test cannot leak the answers into the driver, but a held-out set can leak into the model.
In one precise sentence, the held-out score estimates performance on data that looks like the held-out set, and any difference between that data, or that metric, and production shows up as a gap.
Step 1: Rule out leakage and reuse of the held-out set
Data leakage makes the held-out number too high before the model ships. The scikit-learn documentation says leakage gives overly optimistic performance estimates and poorer performance on novel data in production, and that test data should never be used to make choices about the model.
The common form is preprocessing. If you compute a mean for scaling over all rows, information from the test rows influences the model. The same page notes the risk applies to almost all transformations, including StandardScaler and SimpleImputer, and recommends a Pipeline.
Another form is a feature that only exists after the outcome. Suppose it is present in 80% of held-out rows and makes the model right every time, with accuracy 0.65 otherwise (illustrative numbers). Held-out accuracy is 0.93, while online, where the feature does not exist yet, it is 0.65.1
The last form is reuse of the held-out set. Tuning or picking checkpoints against the same test set many times keeps an optimistic best score, because you selected it for being high. A random split of time-ordered data has a similar effect. The checks are a fresh test set used once, and a time-based split for ordered data.
You now have the first check: was anything fitted, selected or recorded using the test data?
Step 2: Pipeline and data differences cause training-serving skew
Three things cause training-serving skew, according to Google's Rules of Machine Learning: a discrepancy in how the training and serving pipelines handle data, a change in the data between training and serving, and a feedback loop between the model and the algorithm. Rule #31 adds that a table joined at training time and again at serving time may change in between.
The danger is silence. Google Cloud describes a refactor of the serving stack that pins a feature to -1: the model keeps producing outputs with no error, only at lower accuracy.
The fix is to log the serving features and compare them with the training distribution per time window. In the Vertex AI monitoring described in Google Cloud's post, the default window was 24 hours and could be set to any value above 1 hour. You now know where to look when the held-out data is clean but live inputs are not.
Step 3: The traffic mix differs from the held-out mix
The traffic mix is the share of each kind of request in a dataset, and it differs between a held-out set and live traffic. A model can be unchanged and still score lower online because live traffic has more hard requests. Take two segments, easy and hard (illustrative numbers).1
Accuracy is 0.92 on the held-out mix and 0.875 on the online mix, because the same segment accuracies of 0.95 and 0.80 are averaged with different shares. Reweighting segments by their online share gives 0.875. This is importance weighting: a sample kept with 30% probability gets weight 10/3 (Rule #30). Online logs add selection bias, because you only see outcomes for what the system chose to show. Reweighting fixes the estimate, not the product: to raise online performance, improve the model on the hard segment, or resample the training data toward the online mix.
| Segment | Held-out share | Online share | Accuracy | Weight |
|---|---|---|---|---|
| Easy | 0.8 | 0.5 | 0.95 | 0.625 |
| Hard | 0.2 | 0.5 | 0.80 | 2.5 |
Text description of the diagram
- Accuracy is 0.92 on the held-out mix and 0.875 on the online mix.
- Reweighting segments by their online share gives 0.875.
Step 4: The metric is only a proxy for the goal
A proxy metric is a number you can compute cheaply before launch, such as accuracy, that stands in for the outcome you care about, such as click-through rate (CTR) or revenue. Held-out accuracy can rise while the business metric stays flat, because the model improved on cases users rarely meet or on a quantity users do not value. Only an online experiment measures the outcome itself.
For language-model products the same two failures appear. A judge model can score outputs differently from users, and a golden set built from old or hand-picked prompts can miss live traffic.
Check the split first, then the serving features, then the traffic mix, then the metric. Each check removes one cause of a high held-out score.
Step-by-step description
- Rule out leakage in the split.
- Compare serving features with training.
- Reweight segments by online share.
- Confirm with an online experiment.
- Check the split first, then the serving features, then the traffic mix, then the metric.
- Each check removes one cause of a high held-out score.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
Your model scores higher on the held-out set than online. Why, and what do you do?
I treat the held-out score as an estimate that can be wrong in four ways. First, leakage: I check that scalers, imputers and feature selection were fit on training data only, that no feature is recorded after the outcome, and that I did not select on the test set repeatedly. Second, skew: I log serving features and compare their distributions with training, because a pipeline bug changes inputs without raising an error. Third, mix: I reweight held-out segments by their online share. Fourth, the metric: accuracy may not move the business metric, so I confirm with an online experiment before trusting any gain. I fix the cheapest cause first, then keep monitoring skew.
What makes it Staff-level: it orders the checks by cost, names the monitoring that stops the gap returning, and refuses to let a held-out gain stand in for an online result.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Causes | "The model overfit." | Leakage, skew, data or traffic change, proxy metric. | Same four, plus repeated selection on the held-out set, feedback loops and selection bias in logs. |
| Diagnosis | Retrain and hope. | One concrete check per cause. | Cheapest check first; compares the gap with noise. |
| Fix | Add more data. | Fit preprocessing on training only; log and compare serving features. | Adds skew monitoring and reweighting to the release process. |
| Follow-up readiness | Cannot say what an online test adds. | Explains proxy versus business metric. | Explains how a gate on every change keeps the gap visible. |
Three variants of this question
Why is my cross-validation score higher than production accuracy?
Cross-validation can be optimistic if preprocessing or feature choices used the test folds, which is leakage (scikit-learn). It can also be accurate for data that no longer matches production.
Held-out ranking quality improved but the online A/B test shows no lift. Why?
The held-out metric is probably a proxy that does not track the online goal, or the improvement is concentrated in requests that live traffic rarely contains. Check the traffic mix and the metric before blaming the experiment.
Our LLM judge scores are high but users complain. Why?
The judge or the prompt set does not match real users. Compare judge scores with human ratings on live samples and rebuild the golden set from real traffic.
Common mistakes
- "The model overfit to the training data." (wrong) A held-out score already penalises that. (right) "Either the held-out set leaked, or I overfit to it by selecting on it repeatedly; I check leakage, reuse, skew, mix and the metric."
- "If serving were broken we would see errors." (wrong) A skewed model keeps producing outputs at lower accuracy. (right) "I monitor serving feature distributions against training."
- "Cross-validation proves production performance." (wrong) Leakage makes it optimistic. (right) "It is an estimate for data like the folds; I confirm online."
- "A better held-out metric means a better product." (wrong) The metric may be a proxy. (right) "I treat it as a filter and decide with an online experiment."
Follow-up questions
How do you detect training-serving skew?
Log the serving features and compare their distribution with the training baseline per time window, using a statistical distance score. In the Vertex AI monitoring that Google Cloud's post describes, the default window was 24 hours.
How do you prevent preprocessing leakage in scikit-learn?
Fit scalers, imputers and feature selection on the training split only, ideally inside a Pipeline so each step runs on the right subset (scikit-learn).
How large must the gap be before you worry?
Larger than noise. With 1,000 held-out examples (illustrative) and accuracy 0.92, the standard error is 0.0086 and a 95% margin is 0.0168, so the 0.045 gap from the traffic-mix example is about 2.7 times that margin.1 This assumes the online estimate rests on far more traffic, so its own error is negligible; otherwise use the standard error of the difference.
What changes for LLM applications?
The causes stay the same, but the metric is often a judge and the sample a golden set, so both need checking against real traffic. An eval platform that gates every change keeps that check routine.
- A held-out score above the online score means the estimate failed; four causes explain most cases.
- Leakage inflates the held-out score; fit preprocessing on training data only.
- Training-serving skew fails silently; log serving features and compare them with training.
- A shifted traffic mix can be corrected by reweighting segments by online share.
- A proxy metric needs an online experiment to confirm that users benefit.
Sources
- Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (page footer: last updated 2025-08-25 UTC), 25 Aug 2025
- Common pitfalls and recommended practices, section Data leakage (scikit-learn 1.9.1) — scikit-learn documentation (version 1.9.1, released September 2026; day not stated), 1 Sep 2026
- Monitor models for training-serving skew with Vertex AI — Google Cloud Blog (byline date), 31 Jul 2021
