In this article
Before you start
The machine learning algorithms overview covers the models used here, and the statistics interview guide covers the underlying ideas.
Why interviewers ask this
A leaked model looks excellent in the notebook and fails in production, so interviewers want to know whether you would catch it before anyone celebrates. The scikit-learn documentation describes the result as overly optimistic estimates, such as cross-validation scores, followed by poorer performance on genuinely new data. Here data leakage means this machine learning problem, not the security meaning of unauthorized exposure of data.
A weak answer says "do not put test rows in the training set." A strong answer also covers preprocessing, tuning, features built from the future, and rows that share a user or a time period. It adds how you would notice: a score that jumps too far after one new feature should trigger an audit before it triggers a celebration.
A Staff-level answer names the structure of the data. When rows are not independent and identically distributed (i.i.d.), a random split can itself leak, because related rows land on both sides. Kaufman et al. note that early accounts of leakage did not cover cases where the i.i.d. assumption is violated. The question is common in data science loops, and a prompt about a jump in a churn model's AUC is one reported by candidates.
Data leakage from first principles
- Data leakage
- Data leakage is the introduction of information about the prediction target that should not legitimately be available to the model at training or prediction time, which makes validation and test scores overly optimistic compared with performance on new data.
- Target leakage
- Target leakage is a kind of data leakage in which a feature contains the answer or is only known after the moment you must predict, such as a cancellation flag used to predict cancellation.
In plain words
Imagine a teacher who writes an exam, then lets a student see the answer key while revising. The student scores well and is not better at the subject. Data leakage is that answer key reaching the model. The analogy breaks in one place: leaks are rarely a whole key. They are often one clue, such as the average of all rows, which includes the rows you later grade.
In one precise sentence, a model must see only what would exist at the moment of the real prediction, and every number used to build features must come from the training rows alone.
The scikit-learn guidance states the rule briefly: test data should never be used to make choices about the model. "Choices" includes more than fitting weights, as the next steps show.
Step 1: Split before you look
Split the data into training and test subsets first, before any preprocessing, before exploring, and before choosing features. The scikit-learn documentation says to always split first, particularly before preprocessing.
Input: the full table. Output: a training part and a held-back part, with a validation part carved from the training part when you tune. A common split is a majority for training and the rest shared by validation and test, though the ratio depends on how much data you have.
You now know where the wall stands. Everything after this step must not look over it. Exploring the full table first, for example plotting label averages by column, already lets test information shape your choices, so even that waits until the split is done.
Step 2: Fit every transformation on training rows only
Fitting a scaler, imputer, principal component analysis (PCA) or feature selector on all rows leaks, because the numbers it learns include the held-back rows. Feature selection is a step that keeps the columns most related to the label, so it uses the label directly, which makes it especially dangerous.
The scikit-learn documentation says never to call fit or fit_transform on test data and notes that the risk applies to almost all transformations, including StandardScaler, SimpleImputer and PCA. Fit on training rows, then only transform the rest.
Input: training rows. Output: learned statistics, applied unchanged to later rows. In the worked example, selecting columns on all rows turns random noise into a test AUC of 0.87, while selecting on training rows gives 0.51.1
Step 3: Tune inside the training data
Hyperparameter tuning leaks when you pick settings by looking at test scores. Each peek is a choice made with test information, and many peeks fit the test set indirectly. Choose settings with validation rows or cross-validation inside the training part, then score the test part once. The best cross-validation score from a grid search is itself optimistic, because the settings were chosen to maximize it; nested cross-validation, or a single untouched test set, gives the honest estimate.
A pipeline makes this safe. The scikit-learn documentation calls it a great way to prevent leakage because the right method runs on the right subset, and it says the pipeline suits cross-validation and hyperparameter tuning. With a pipeline passed to the search, every fold refits scaler and selector on that fold's training rows.
Input: a pipeline and a grid of settings. Output: the best settings, picked without touching the test rows.
Step 4: Respect time, groups and the moment of prediction
Time, groups and duplicates break the random split. Features built from the future leak, and so do rows from one user, patient or session sitting on both sides. Kaufman et al. say simple accounts of leakage do not cover cases where the i.i.d. assumption is violated.
For time-ordered data, train on earlier rows and test on later rows, and build every feature from values known before the prediction moment. For grouped data, keep each group wholly on one side. For duplicates, deduplicate before splitting.
Input: rows with timestamps or group ids. Output: a split that matches how the model will be used. A drift check, as in distribution shift, is the related question about the future differing from the past.
Step-by-step description
- Hold back the test rows before looking.
- Scalers and selectors learn from training rows.
- A pipeline refits each step per fold.
- Use only what is known at prediction time.
- Split the data into training and test subsets first, before any preprocessing.
- Fit every transformation on training rows only.
- Tune inside the training data with a pipeline.
- Respect time and groups so features use only what is known at prediction time.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Two small synthetic experiments show the size of a leak. Both are illustrative, built with random numbers, and averaged over 30 seeds.1
In the first, 100 rows have 2,000 random columns and a random 0 or 1 label, so nothing can be learned and a fair score is 0.5. Keeping the 10 columns most related to the label, then splitting 50 and 50, gives a mean test AUC of 0.87. Keeping the 10 columns using the 50 training rows only gives 0.51.1
In the second, a daily series of 3,000 days has a label that marks days above the median. A 7-day average that includes the label day reaches a mean AUC of 0.72. The same average shifted by one day, using only earlier days, gives 0.61.1 The leaky window includes the thing it predicts, and the gap is the leak: 0.36 AUC points for the noise columns and 0.11 for the window.1 In this second experiment the 7-day average is used directly as the score, with no model and no train/test split, so its AUC is not a test AUC.
| Experiment (illustrative, synthetic) | Leaky version, mean AUC | Clean version, mean AUC |
|---|---|---|
| Select 10 of 2,000 noise columns | 0.87 | 0.51 |
| 7-day average on a time series | 0.72 | 0.61 |
Text description of the diagram
- Selecting columns using all rows gives a mean test AUC of 0.87 for noise.
- Selecting using training rows gives 0.51.
- A 7-day average including the label day gives 0.72 against 0.61 when lagged.
The lesson for the interview prompt in which a churn model's AUC goes from an illustrative 0.71 to 0.93 after one rolling feature: a gain that large from one column is a symptom to audit. Check what days the window covers, whether it includes the label day, and when each value would be known in production.
Trade-offs and edge cases
Leakage is a research problem, not a beginner slip. A survey in the journal Patterns found 17 fields where leakage has been reported, collectively affecting 294 papers, and proposes a taxonomy of eight types of leakage. In one case study there, once errors were corrected, complex models did not perform substantively better than decades-old logistic regression.
Edge cases:
- Keeping the sets apart costs data. Holding out validation and test rows leaves fewer rows for training; cross-validation inside the training part recovers some of that.
- Pipelines do not fix everything. A pipeline stops fitting leaks, not a feature that is known only after the event.
- Time order shrinks your options. Forward splits give fewer folds and the earliest fold trains on little data. Walk-forward backtesting, which trains on one window, tests on the next period and then rolls forward, is the usual form.
- Shared entities leak quietly. Rows from one user are alike, so scores inflate unless groups stay together.
- Language models. The same idea applies to large language model (LLM) evaluation: a benchmark score is inflated if its questions appeared in the training text. Check for overlap before trusting it.
Leakage is detected by its symptoms, because no single test names it. Warning signs are one feature that dominates the importance ranking or scores a near-perfect AUC on its own, a score far above a simple baseline or above what the domain makes plausible, a score that collapses on a time-ordered or grouped holdout or in production, and duplicate or near-duplicate rows sitting across splits. The remedy is to drop the suspect feature and re-score; if the score falls back, that feature leaked.
Text description of the diagram
- Fitting a scaler on all rows leaks, because the numbers it learns include the held-back rows.
- Fitting a transformation on training rows only keeps held-back rows out of what it learns.
- Time leakage uses a random split on a series; split forward in time instead.
- Group leakage places the same user on both sides; split by group.
Good feature engineering respects the same rule: encoders, imputers and aggregates are fitted on the training part. For event data over time, see also anomaly detection on a time series.
How to answer it in an interview
What is data leakage and how do you prevent it?
Data leakage is when information the model would not have at prediction time reaches training, so my validation score is optimistic. It shows up as test rows in training, but more often as a scaler or feature selector fitted on all the data, tuning on the test set, or features built from the future. I split first, fit every transformation inside a pipeline on training rows only, tune with cross-validation inside the training part, and split by time or group when rows are related. A sudden score jump makes me audit features first.
The spoken structure: define it, name the three common sources, give the prevention order, then say what you do when a score looks too good.
Common mistakes
- "I scaled the data, then split it." (wrong) The scaler learned the test rows' mean. (right) "I split first and fit the scaler on the training rows."
- "Cross-validation is safe, so I selected features first." (wrong) Selection used every fold's labels. (right) "Selection sits inside the pipeline, refitted per fold."
- "I picked the best settings on the test score." (wrong) That is a choice made with test data. (right) "I tuned on validation folds and scored the test set once."
- "A random split is standard." (wrong) It mixes past and future rows. (right) "For time data I train on earlier rows and test on later ones."
- "The score went up a lot, so the feature is great." (wrong) A large jump needs an audit. (right) "I first check when that feature is known in production."
Follow-up questions
How do you split the train data into train, test and validation?
Split off the test set first and leave it alone. Carve a validation set from the training part, or use cross-validation inside it, to choose settings. Split by time for ordered data and by group when rows share a user or a patient.
How would you compute time-window aggregate features for users without leaking future information?
For each user and each prediction time, compute the window only from events before that time, using a one-sided "as of" boundary. Use the same code and the same as-of rule at training and at serving so the two match, and never use the label day inside the window.
Your churn model's AUC jumps from 0.71 to 0.93 after adding a 7-day rolling feature. What now?
Suspect leakage first. Check whether the window includes the label period, drop the feature and confirm the score returns, then rebuild it lagged and re-score on a time-ordered split. Only a gain that survives a clean split is real.
When a model reaches saturation and you cannot change architecture or data, what else can you tune?
Tune the loss, regularization, learning rate schedule, class weights, decision threshold and ensembling, using validation folds only. Keep the test set untouched, since choices made on it leak.
How do you keep a held-out set untouched in a take-home assignment?
Split off the test portion before any exploration, set it aside, and score it once at the end, saying so in your write-up. If the assignment scores a hidden test set, do not tune on its feedback; use cross-validation inside the training data instead.
Can data leakage happen in a time series?
Yes, and it is the harder case. A random split places future rows in training, and a window that includes the target day leaks. Use forward-in-time splits and lagged features.
- Data leakage is information the model should not have at training or prediction time, and it inflates scores.
- Split first, then fit every transformation on training rows only.
- Tune with validation folds inside the training part; use a pipeline.
- Time, groups and duplicates need their own split rules.
- A large score jump from one feature is a reason to audit, not to celebrate.
Sources
- Common pitfalls and recommended practices: data leakage — scikit-learn documentation (stable docs, version 1.9.1; the page is undated, so the date is the read date), 26 Oct 2026
- Leakage and the reproducibility crisis in machine-learning-based science — Patterns, Cell Press (Kapoor and Narayanan), 4 Aug 2023
- Leakage in data mining: formulation, detection, and avoidance — ACM Transactions on Knowledge Discovery from Data (Kaufman, Rosset, Perlich, Stitelman), 1 Dec 2012
