In this article
Before you start
For the wider topic see machine learning operations (MLOps) interview questions, and for the models themselves see machine learning algorithms.
Why interviewers ask this
A model can pass every test before launch and still lose quality in production, with no error message and no crash. Interviewers ask about skew because it separates candidates who treat a model as a saved file from those who treat it as a pipeline whose inputs must match at both ends.
A weak answer says "retrain the model" or "monitor drift." Neither names a cause. A strong answer names where two paths can disagree (code, null handling, units, joins, freshness), says how to prove the disagreement with a replay, and says how to stop it coming back. A Staff-level answer adds the cost: shared code limits how each side can be optimized, so you decide which features need strict parity and which can tolerate small differences. A strong answer also keeps skew apart from drift, which is a separate problem with a separate fix.
Training-serving skew from first principles
- Population stability index
- The population stability index (PSI) is a number that measures how far one distribution has moved from another by comparing the share of values in each bin. A value near zero means the two distributions match, and a larger value means a bigger shift.
In plain words
Imagine a recipe cooked in two kitchens. The test kitchen measures flour in grams and the restaurant kitchen measures it in cups. The recipe card is identical, yet the dishes differ. Nobody complains about the recipe, because the card never changed; the difference lives in how each kitchen reads it. The analogy breaks in one place: in a model, nobody tastes the dish, so the difference stays silent until a metric moves, and by then users have already seen worse results.
In one precise sentence, pipeline skew exists whenever the function that turns a raw record into model inputs is not the same function in training and in serving. Every other cause on this page, from null handling to joins, is a special case of that sentence.
Step 1: Replay the same rows through both paths
Replaying means taking a sample of raw records, running the training feature code and the serving feature code on each, and comparing the outputs column by column. The step answers "which feature differs, and on how many rows?" Input: raw rows. Output: a table of per-feature disagreement.
Google, Rules of Machine Learning asks for the same basic property: the model in training should give the same score as the model in serving. It also advises tests for the code that creates examples in both places, and checking pipeline statistics against the same data processed elsewhere. The guide lists three causes of skew: a discrepancy in how the training and serving pipelines handle data, a change in the data between training and serving, and a feedback loop between the model and the algorithm.
A replay mismatch means a pipeline bug. If the replay agrees but production numbers still moved, the world changed, which drift monitoring alone cannot tell you.
Typical causes are a missing value filled differently, a unit mismatch, a feature rewritten in a second language, and a stale lookup table. The next step removes the cause.
Step 2: Give every feature one definition
The fix is one definition per feature that both paths use. There are two common ways. Either share the code, so the same function runs in the training job and the service, often through a feature store. Or log the features the service actually used at prediction time and train on those logged values; Google, Rules of Machine Learning (Rule #29) advises to "save the set of features used at serving time, and pipe those features to a log to use them at training time". The cost is that a new feature has no training data until enough logs have accrued.
Joins need care. A training join that uses the latest value of a table, rather than the value known at prediction time, leaks the future into training. A point-in-time join prevents that. Freshness matters too: a feature computed by a nightly batch job in training but from a live stream in serving differs by construction; see batch, streaming and real-time inference. Input: the mismatch found in Step 1. Output: a feature that both paths compute identically.
Step 3: Monitor feature distributions in production
Monitoring means comparing each feature's distribution in serving against the training baseline on a schedule, and alerting when they separate. The population stability index is one way to turn that comparison into a single number per feature. Re-running the Step 1 replay on a sample of logged production rows catches regressions that a one-time check misses, for example after someone edits one path.
Monitoring does not tell you why the numbers moved. Two checks answer different questions: compare per-row values in the replay to find a feature computed differently, and compare distributions in monitoring to find a feature whose values have moved. Usage differs: Google's guide counts a data change between training and serving as skew, while many teams keep "skew" for pipeline mismatches and call change over time "drift"; the replay separates the two. The fixes differ, and model drift covers the second case. The same habit applies to prompt-based systems built on a large language model (LLM); see LLM observability.
Step-by-step description
- Run the same raw row through both paths and compare.
- Give each feature one shared definition.
- Watch production distributions against training.
- Replay the same raw rows through the training path and the serving path.
- Compare the features column by column to find the mismatch.
- Give each feature one shared definition.
- Monitor feature distributions in production against the training baseline.
Worked example: one null-fill mismatch, measured
A small synthetic example (illustrative data) shows the size of the effect. A model predicts a yes or no outcome from two features, one of them days since the customer's last order. About 20% of rows have no order history. The training path fills the gap with the training mean of 20.03 days. A buggy serving path fills it with 0, which reads as "ordered today."1
The model scores 72.83% accuracy through the training path and 71.06% through the buggy serving path, a drop of 1.77 points; its log loss rises from 0.5313 to 0.5969.1 The feature differs on 19.05% of rows, and its mean falls from 19.96 to 16.14 days.1
| Check | Training path | Buggy serving path | Fixed serving path |
|---|---|---|---|
| Accuracy on 20,000 held-out rows | 72.83% | 71.06% | 72.83% |
| Rows where the feature differs | 0% | 19.05% | 0% |
| PSI against the training path | 0 | 0.6067 | 0 |
The population stability index is 0.6067 for the buggy path and 0 for the fixed one, so the distribution check is far louder than the accuracy metric.1
Text description of the diagram
- The population stability index is 0.6067 for the buggy serving path.
- The population stability index is 0 for the fixed serving path.
- Accuracy drops only 1.77 points on the buggy path.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you handle training-serving skew?
Training-serving skew means the features the model sees in production differ from the ones it trained on. I handle it in three moves. First, I replay the same raw rows through the training path and the serving path and diff every feature, because that locates the exact mismatch, often a null fill, a unit or a join.
Second, I remove the cause by giving each feature one definition used by both paths, or by logging serving-time features, which makes a new feature wait for logs to accrue. Third, I monitor feature distributions in production against the training baseline and alert on a separation. The trade-off is that sharing code constrains latency choices, so I accept a slower shared path only where parity matters.
What makes it Staff-level: it proves the cause before fixing it, names a prevention and a detection mechanism, and states the cost of the fix.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Diagnosis | "Retrain the model." | Replays rows through both paths and diffs features. | Also separates skew from drift and leakage. |
| Fix | "Be careful with the code." | One shared feature definition. | Shared code or logged features, chosen per feature by latency need. |
| Detection | "Watch accuracy." | Compares feature distributions to the training baseline. | Adds scheduled replays and alerts tuned on a known-good period. |
| Trade-offs | None named. | Names extra engineering cost. | Names latency, freshness and ownership costs. |
| Communication | Jargon without a cause. | States cause, fix, check in order. | Gives a concrete example of a mismatch and its size. |
Three variants of this question
How do you detect training-serving skew before users notice?
Replay a sample of raw rows through both feature paths before release and compare the outputs, then compare production feature distributions with the training baseline on a schedule. The replay finds code differences; the distribution check finds data differences.
What is the difference between training-serving skew and data drift?
Pipeline skew is a mismatch between the training and serving pipelines that is present from launch. Drift is the data changing over time; see model drift. A replay of the same rows through both paths separates them, because a replay mismatch means pipeline skew.
How can a feature store prevent training-serving skew?
A feature store keeps one definition of each feature and serves it to both training and inference, and many feature stores support point-in-time joins for training data. It does not remove skew by itself if a team still computes some features outside it.
Common mistakes
- "The holdout score is good, so the model is fine in production." (wrong) The holdout set passed through the training pipeline, not the serving one. (right) "I also score the model through the serving path and compare."
- "Skew and drift are the same thing." (wrong) They have different causes and fixes. (right) "Pipeline skew is present from day one; drift is change over time."
- "I'll rewrite the feature in the service's language." (wrong) A second implementation is a second chance to disagree. (right) "I share the code or log serving-time features."
- "Accuracy is stable, so there is no skew." (wrong) In the worked example a 1.77-point drop hid a large distribution shift. (right) "I monitor feature distributions, not only the final metric."
Follow-up questions
Why does my model score well on the holdout set but badly in production?
The holdout set was built by the training pipeline, so it cannot reveal a serving mismatch. Replay the same rows through the serving path, diff the features, and check whether production data has changed since training.
What do you do when you cannot reuse the same code in training and serving?
Log the features the service used at prediction time and train on the logged values, and keep a parity test that runs both implementations on the same rows. See designing a feature store for the shared-definition route.
How do you set an alert threshold for a skew metric?
There is no universal number. Measure the metric on a period known to be healthy, set the alert above its normal variation, and check that the alert fires on a replay of a past incident.
Does skew apply to large language model prompts?
Yes, in the same way: a prompt template or retrieval step that differs between evaluation and production changes the model's input. Tracing the exact production prompt, as in LLM observability, is the equivalent of logging serving features.
- Pipeline skew is present from launch and drift is change over time; the replay tells them apart.
- Replay the same raw rows through both paths and diff every feature to find the cause.
- Give each feature one definition, shared or logged, and use point-in-time joins for training data.
- Monitor feature distributions, because a small accuracy drop can hide a large input shift.
- Name the trade-off: shared code limits latency and ownership choices.
Sources
- Rules of Machine Learning: Best Practices for ML Engineering — Google for Developers (undated page; read 2026-10-26), 26 Oct 2026
- Rules of Machine Learning: Rule 29, save the features used at serving time — Google for Developers (section of the same guide; read 2026-10-26), 26 Oct 2026
