Trained in January, bad in July: model drift explained

In this article

1. Before you start
2. Why interviewers ask this
3. Model drift from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Drift is one of the core machine learning operations (MLOps) topics, and this page assumes the basics from machine learning algorithms and how they work.

Why interviewers ask this

Models that fail quietly in production cost more than models that fail loudly, so interviewers use drift to see whether a candidate thinks past training. The prompts below are reported by candidates in machine learning system-design rounds: how to handle data drift or model decay, how to monitor a model in real time, and how to evaluate a model after deployment.

A weak answer says "retrain every month." A strong answer separates the two kinds of drift, says how each is detected, notes that labels arrive late, and treats an alert as a reason to investigate, not as proof of failure. A Staff-level answer adds the response path (investigate, roll back or retrain), who owns the thresholds, and what the retraining data should contain (answered under "What data do you retrain on?" below).

Model drift from first principles

Concept drift

Concept drift is a change in the relationship between a model's inputs and the target it predicts, so the same input now deserves a different answer. The inputs can look unchanged while the correct outputs move.

Kolmogorov-Smirnov test

The Kolmogorov-Smirnov (KS) test is a statistical test that compares two samples of a continuous feature. It reports D, the largest gap between their cumulative distributions, and a p-value; a small p-value means a gap this large would be unlikely if both samples came from the same distribution.

In plain words

Think of a student. The student studies one winter's exam papers. Then the student sits an exam in summer. The questions have changed. The student remembers the winter answers well but cannot tell. The analogy breaks in one place. A student can feel lost. A model gives no reliable signal of confusion: under shift its confidence often stays high. Ovadia et al. (2019) report that predictive uncertainty degrades under dataset shift.

In one precise sentence, a batch-trained model is a fixed function of its inputs. A change in the world that it has not been retrained on shows up as lost quality.

Step 1: A model is a frozen snapshot

A deployed model keeps using what it learned during training. Evidently AI's concept-drift guide states that models do not automatically notice new patterns, and that they keep predicting from what they previously learned until someone explicitly retrains or updates them.

Input: the training data, here only January. Output: a rule that fits January. Nothing in the rule says "I have not seen summer." Retraining is what changes the rule. The one exception is a model built to learn continuously, covered under the edge cases below. In the simulation later on this page, the model is a straight line fitted once on January days and never touched again. You now know why decay is the default, not a bug: standing still is what a batch-trained model does while the world keeps moving.

Step 2: The world moves in two ways

Two things in the world can move: the inputs, and the relationship between inputs and the target. Evidently AI separates data drift (the model inputs change) from prediction drift (the model outputs change); prediction drift is a symptom you can observe, not a third way the world moves. Concept drift is the harder case. The Snowflake monitoring guide describes it as the relationship between inputs and the target changing, so the same data now means something different.

Training-serving skew is a different problem again: it is a mismatch between the training data and the production data, while data drift is a shift in production inputs over time (Evidently AI). Both kinds of drift can lower quality, and they often arrive together. The inputs are what you can see; the relationship is only visible through labels.

1 · Frozen snapshot2 · World moves3 · Compare4 · RespondReplay

Step-by-step description

1. The model learns January and then stands still.
2. Inputs or the input-to-target link change.
3. Production inputs are compared with a baseline.
4. Investigate, roll back or retrain.

• A model keeps predicting from its training snapshot until it is retrained.
• Drift is found by comparing production inputs with a baseline.
• The response is to investigate, roll back or retrain.

Step 3: A drift alert comes before the labels do

Detecting drift means comparing current production inputs with a baseline, which is typically the training data, a validation window or a recent stable period in production (Snowflake). For a continuous feature, groundcover suggests the KS test; for category frequencies, the chi-squared test. The population stability index (PSI) is a score that cuts a feature into bins taken from the baseline and sums, over the bins, the gap between current and baseline share times the log of their ratio; zero means no shift.

Why bother with inputs? Labels can arrive late: in some domains the true outcome takes weeks or months to be known, so the error metric lags behind the world. Evidently AI's data-drift guide describes techniques for the case where ground truth is absent, such as watching prediction drift and data drift as proxies. So the input alert is the early signal, and the error metric is the confirmation that comes later.

Step 4: The response is investigate, roll back or retrain

Teams define reference distributions, set thresholds for tolerable movement, and establish response paths for investigation, rollback or retraining when signals go out of bounds (Snowflake). An alert starts the investigation; it does not decide the outcome.

Some movement is normal: seasons, launches and ordinary changes in customer behavior all shift production data, so ask whether a shift is large, persistent or concentrated in one segment. A significant test does not prove performance has declined (groundcover). If the investigation finds a real loss, you choose between rolling back to an older model and retraining on recent data. Sources differ on cadence, so state your own rule and its reason.

Worked example

The example is a simulation, not a measurement: an illustrative demand model for a temperate city. The model is a straight line fitted on 2,000 January days, with temperature as the only input; the true demand is a curve that rises on both cold and hot days. Each later month draws 500 days with that month's illustrative mean temperature.¹ This is pure data drift: the true curve never changes, only the temperatures do, so an input test can see it early; under concept drift with unchanged inputs the same KS test would stay silent.

For each month the script computes the mean absolute error (MAE), which is the average size of the gap between prediction and truth, plus the KS statistic D against the January days and the PSI.¹ The alert rule is an illustrative p-value below 0.01.

The first KS alert is February (p about 1.9e-11) while the February error, 2.48, is no worse than January's 2.67.¹ The error first exceeds twice January's in April (6.70). With an illustrative label delay of four weeks, April's error is only confirmed around May, so the input alert leads the confirmed error by about three months.¹ The simulated error is 2.67 in January and 24.88 in July, about nine times January's, and the error is back to 2.67 in December.¹

December is the lesson about false alarms. With 500 samples the KS test flags a tiny shift (D 0.094, p 0.0016), yet the MAE is 2.67, equal to January, and PSI is only 0.07.¹ Large samples make small shifts significant; look at the size of the shift as well as its p-value.

Text description of the diagram

• The simulated error is 2.67 in January and 24.88 in July.
• The error is back to 2.67 in December.

Trade-offs and edge cases

No single check finds every kind of drift (groundcover). Input checks miss concept drift that leaves the inputs unchanged, and label-based checks are late. The practical design uses both: input and prediction drift for speed, error metrics for truth.

Text description of the diagram

• Data drift changes the inputs and can be seen without labels.
• Concept drift changes the relationship between inputs and the target.

Edge cases to name in an interview:

• Seasonality is not an incident. If the drift is a calendar cycle, as in the simulation, a model trained on one season is the flaw; train on at least a full cycle and compare with the same period last year.
• Fast and slow decay. Decay can take hours or days when user preferences move fast, and months or years for stable processes (Evidently AI), so monitoring frequency follows the domain.
• Fraud-style ambiguity. A rise in false positives could come from a popular sale item or from a real attack, which needs a different response, so check which segment moved before you act.
• Skew is not drift. A gap that exists from day one is a pipeline bug; a feature store that serves the same feature code in training and serving helps.
• Many features, many tests: with 100 features each tested at p < 0.01, about one false alert per check is expected by chance (a computed example: 100 × 0.01 = 1, and a 63% chance of at least one).¹ Correct for multiple tests, or alert on effect size and persistence.
• Online learning: a model updated incrementally does adapt, and Gama et al. (2014) survey such adaptive methods. The frozen-snapshot rule applies to batch-trained models.

What data do you retrain on?

Retrain on data that matches what the model will see next, and prove it on a recent window before you promote it. You have three common choices. A sliding window keeps only the last few months, so patterns that no longer hold drop out, but rare events and old seasons drop out too. All history keeps every pattern but lets stale ones dilute new ones. Recency-weighted samples keep all history and give recent rows more weight, a middle path.

Gama et al. (2014) survey windowing and weighting methods for concept drift, including adaptive windows. Whichever you pick, include at least one full seasonal cycle, or the next season surprises the model again; the January-only training set in the simulation is the failure case. Then hold out the most recent window, which the new model never trained on, and compare the new and current models there. Promote only if the new one wins. Treat this as design advice, not a sourced standard.

How fast you can compute these checks depends on the serving style; see batch, streaming and real-time inference. For language models, the same idea reappears as large language model (LLM) observability.

How to answer it in an interview

The spoken structure: say it decays by default, name how you detect it with and without labels, then the response path and the false-alarm caveat.

Common mistakes

• "Our test score tells us how good the model is in production." (wrong) The model keeps predicting from its training snapshot and does not adapt on its own. (right) "The test score describes the snapshot; production quality needs monitoring."
• "Data drift and concept drift are the same thing." (wrong) One is the inputs changing, the other is the input-to-target relationship changing. (right) "Data drift moves the inputs; concept drift changes what the right answer is."
• "The alert fired, so the model is broken." (wrong) A significant test does not prove the performance declined. (right) "The alert opens an investigation; I confirm with an error metric."
• "No input drift, so we are fine." (wrong) Concept drift can occur with unchanged inputs. (right) "I also watch predictions and the error metric once labels land."
• "Retrain every month and stop there." (wrong) A fixed schedule alone ignores both the cause and the cost. (right) "I retrain when a monitored signal crosses a threshold, with a schedule as a backstop, and I justify the cadence."

Follow-up questions

What if labels take six months to arrive?

Lean on input and prediction drift as early signals, and judge quality with the delayed error metric on cohorts whose labels have matured. State the age of every number you report. Keep a rollback path ready, because you can detect a possible problem long before you can measure its cost.

How would you design a system to monitor the performance of a machine learning model in real time?

Log inputs, predictions and later labels with a shared request id; compute drift scores per feature on a window; chart prediction distributions; and join labels when they arrive. As design advice, not a sourced standard: alert on persistent shifts rather than single points to limit noise, and choose the window by how fast the domain moves.

How would you evaluate the performance of a machine learning model in a production environment?

Use the true error metric once labels arrive, and while waiting, use proxies such as data drift and prediction drift. Report the label delay so readers know how old the number is.

What is the difference between data drift and concept drift?

Data drift is a change in input distribution; concept drift is a change in the relationship between inputs and the target. Data drift can be a symptom of concept drift, but not always.

Does a drift alert mean the model is broken?

No. Seasonal movement is expected, and a significant test does not prove the model got worse; the December row in the worked example shows an alert with no loss. Check the error metric and the size of the shift.

Sources

1. What is concept drift in ML, and how to detect and address it — Evidently AI (ML in production guide; page undated, accessed 2026-10-26), 26 Oct 2026
2. What is data drift in ML, and how to detect and handle it — Evidently AI (ML in production guide; page undated, accessed 2026-10-26), 26 Oct 2026
3. Model Monitoring: How to Catch ML Model Decay Before It Affects Production — Snowflake (page undated, accessed 2026-10-26), 26 Oct 2026
4. What Is Model Drift? Types, Detection & Fixes — groundcover (page undated, accessed 2026-10-26), 26 Oct 2026
5. A survey on concept drift adaptation — ACM Computing Surveys 46(4) (Gama et al.), 1 Mar 2014
6. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (arXiv 1906.02530) — arXiv (Ovadia et al.), 6 Jun 2019