What is distribution shift, and how do you detect it?

In this article

1. Before you start
2. Why interviewers ask this
3. Distribution shift from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the classical machine learning learning path.

Why interviewers ask this

Distribution shift is a common reason a model that looked strong before launch gets worse in production, and interviewers ask about it to see whether you think past the training metric. The question separates candidates who say "retrain on new data" from candidates who first ask what changed.

A weak answer names a drift test and stops. A strong answer separates the three kinds of shift, says that watching inputs alone can be misleading in both directions, says how labels arrive late in most products, and links the response to the cause. It is reported in interviews at AI labs and large tech companies, often as "the pre-deployment metrics were good but production dropped, what do you do?", so prepare an ordered investigation.

Distribution shift from first principles

Distribution shift

Distribution shift is a change between the joint distribution of inputs and labels a model was trained on (how often each combination of input values and label occurs together) and the one it meets in production, which can lower the model's accuracy without any change to the model itself.

Covariate shift

Covariate shift is a kind of distribution shift in which the distribution of the inputs changes while the rule linking inputs to labels stays the same.

Label shift

Label shift is a kind of distribution shift in which the share of each label changes while the inputs of each label look the same.

In plain words

Think of a weather forecaster who learned from ten years of coastal towns and is then asked about a mountain village. If the village simply has colder days than anything in her notes, the inputs changed. If rain is suddenly more common everywhere, the label mix changed. If the same cloud pattern now means something different because of a changed climate, the rule changed. A forecaster knows she moved, but a model gets no warning, which is why Rabanser et al. (2019) say machine learning systems tend to fail silently.

In one precise sentence: a model assumes its production data comes from the training distribution, and shift is any break in that assumption. Usage of "concept drift" differs. Lu et al. (2018, arXiv 2020) define it broadly, as any change over time in the joint distribution of inputs and labels, and call an input-only change "virtual drift" and a changed input-to-label rule "actual drift". This page, like most interview usage, reserves concept drift for a changed rule, their actual drift, and calls an input-only change covariate shift or data drift.

Step 1: A model assumes production looks like training

A model learns a rule from examples and applies it to new ones, so it is only trustworthy where the new examples resemble the old. The input to this step is a trained model; the output is the assumption it silently carries.

A pre-deployment test set drawn from the same period and source as the training set measures accuracy under that assumption. It cannot tell you what happens when the assumption breaks. In production, three things can change: the inputs, the share of each label, or the rule itself. You now know what to look for, and the next step names each change.

Step 2: Three things can change

Covariate shift, label shift and concept drift are the three kinds, and each breaks the assumption in a different place. Covariate shift changes the inputs, for example a new customer segment arriving. Label shift changes how common each outcome is, for example fraud becoming rarer. Concept drift changes the rule, for example a spam pattern that now means the opposite.

The kinds matter because they need different responses. Reweighting examples can fix covariate shift when the rule is unchanged, a corrected threshold can fix label shift, and only fresh labelled data can fix concept drift. Terminology also varies: practitioners say "data drift" for input changes and "model drift" for any decline, while Rabanser et al. use dataset shift for the whole family.

Step 3: Detection can fire without harm and stay silent during harm

An input test measures how different two samples look, not whether the model got worse. The Kolmogorov-Smirnov (KS) test compares one feature in two samples under the null hypothesis, the starting assumption of no difference, that both samples come from the same distribution. Its KS statistic is the largest gap between the two samples' cumulative shares, from 0 for identical samples to 1 for samples that do not overlap. Its p-value is how likely a gap that large would be if nothing had changed, so below 0.05 is the usual signal of a shift and 0.61 means no visible difference.

In the worked example below, the population stability index (PSI), a score of how much one distribution's bin shares moved from another's, was 3.48 for the shifted input while accuracy was 98.8%, unchanged.¹ Under concept drift, the KS p-value was 0.61 and the PSI 0.0044, with accuracy at 65.5%.¹ Detection that watches only inputs misses concept drift, so watch inputs and outcomes together, and treat an input alert as a reason to look, not as proof of harm.

Step 4: The response follows the cause

Retraining is one response, and it helps only when the new data carries the new rule. Diagnose first: check the pipeline for a bug such as a changed unit or a missing field, check which segment changed, then decide whether to reweight, fix a threshold or retrain on recent labelled data.

The input to this step is the diagnosis and the output is the smallest fix that works. A retrain trigger should be a sustained drop in a labelled metric on enough fresh labels, with the cause diagnosed. An input alert on the features that matter is an early warning that starts the investigation; it is not a required condition, because concept drift leaves the inputs unchanged. Lu et al. (2018) frame the work as three parts: detecting drift, understanding it and adapting to it.

1 · Compare the data2 · Name the shift3 · Detect both sides4 · Respond by causeReplay

Step-by-step description

1. A model assumes production looks like training.
2. Inputs, label mix or the rule changed.
3. Watch inputs and outcomes together.
4. Fix, reweight or retrain on recent data.

• A model assumes production data looks like its training data.
• Shift can change the inputs, the label mix or the rule linking them.
• Detection that watches only inputs misses concept drift.
• The response depends on the cause: fix a pipeline bug, reweight, or retrain on recent data.

Worked example

A small simulation separates the three kinds. It trains a random forest (an ensemble of many decision trees that vote) on 4,000 synthetic rows with two features, then scores four production sets of 4,000 rows each: no shift, covariate shift (the first feature's mean moved by 2.0), label shift (positives resampled to 10% of rows) and concept drift (the labelling rule changed so the second feature's weight in the rule changed sign). The data is synthetic and the numbers are illustrative, not measurements of a real system.¹

Accuracy is 98.75% with no shift, 98.80% under covariate shift, 98.68% under label shift and 65.52% under concept drift.¹ The covariate shift moved the first feature a long way (KS statistic 0.68, PSI 3.48) yet did not lower accuracy, because the forest still handled that region of the input space. Label shift left accuracy flat but cut precision from 0.99 to 0.89, because positives became rare. Concept drift left the inputs untouched (KS p-value 0.61 for the first feature) and ruined accuracy.

Text description of the diagram

• Accuracy is 98.75% with no shift and 98.8% under covariate shift.
• Accuracy is 98.68% under label shift and 65.52% under concept drift.

Trade-offs and edge cases

Every detection method trades false alarms against missed shifts, and the right balance depends on how fast labels arrive. Four cases matter.

First, labels arrive late. In many products the true outcome is known days or months later, so input and output monitoring come first and labelled accuracy confirms later. Second, sample size changes what tests say. With many rows, a two-sample test flags tiny, harmless differences; Evidently's documentation (read October 2026) uses, for numerical columns, a KS test with a 0.05 p-value for reference sets up to 1,000 observations and a distance with a 0.1 threshold for larger ones. Third, many features means many tests, and a small rate of false alarms multiplies. Fourth, a shift that breaks only a rare segment does not show in the average, so slice by segment.

Text description of the diagram

• The KS statistic is 0.02 with no shift and 0.68 under covariate shift.
• The KS statistic is 0.28 under label shift and 0.02 under concept drift.

How to answer it in an interview

The spoken structure: rule out a bug, name the kind of shift, say how you detect it, then match the fix to the cause.

Common mistakes

• "Distribution shift means the inputs changed." (wrong) The rule or the label mix can change while inputs look the same. (right) "There are three kinds: covariate, label and concept."
• "A drift alert means the model got worse." (wrong) In the example an alert fired with accuracy unchanged. (right) "An alert is a reason to look, not a verdict."
• "No input alert means no problem." (wrong) Concept drift leaves inputs untouched. (right) "I also track a labelled metric."
• "Always retrain on the newest data." (wrong) A pipeline bug or a threshold shift is cheaper to fix. (right) "I diagnose first, then pick the smallest fix."

Follow-up questions

What is model drift?

Model drift is an informal name for a deployed model getting worse over time. The cause is usually one of the three kinds of shift on this page, a pipeline change, or a change in how outcomes are recorded, so naming the cause matters more than the label. The research papers cited here use the terms dataset shift and concept drift.

Data drift vs concept drift?

Data drift means the inputs changed; concept drift means the rule linking inputs to labels changed. Data drift can be seen without labels by comparing feature distributions, while a changed rule with unchanged inputs needs labelled outcomes to detect. Lu et al. call that case "actual drift" and an input-only change "virtual drift"; see the definitions above for how usage differs (Lu et al. review).

How to detect distribution shift?

Compare production inputs with a reference sample using a two-sample test or a distance, compare output and label rates, and track a labelled metric when labels arrive. Rabanser et al. (2019) found two-sample tests on a reduced representation worked best in their study. Run the tests per segment and on the features that matter.

How to handle model drift?

Diagnose the cause, then act: fix a data pipeline bug, reweight or recalibrate for covariate or label shift, or retrain on recent labelled data for concept drift. Add monitoring so the next shift is caught earlier; the machine learning system design hub and the machine learning operations hub cover the surrounding system.

What if you have millions of rows?

A two-sample test then flags tiny, harmless differences, because the p-value shrinks as the sample grows. Use an effect size such as a distance or PSI with a threshold, and confirm with a labelled metric before acting; the sample-size point in the trade-offs section explains why.

When should you retrain a model?

Retrain when a labelled metric has dropped on enough fresh data and the cause is a changed rule or a new region of inputs that old data does not cover. A schedule is a fallback; a trigger tied to evidence is better. For language-model apps, observability for language-model apps covers the equivalent monitoring.

Sources

1. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — arXiv (Rabanser, Günnemann and Lipton), 29 Oct 2018
2. Learning under Concept Drift: A Review — arXiv (Lu et al.), 13 Apr 2020
3. scipy.stats.ks_2samp, SciPy reference guide — SciPy (date = read date), 24 Oct 2026
4. Data drift, Evidently documentation — Evidently AI (date = read date), 24 Oct 2026