Classical ML

How do you determine if a model is overfitting?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

How do you determine if a model is overfitting? Compare its score on training data with its score on held-out data; a high training score beside a clearly lower held-out score means the model memorized the training set. That comparison only counts if the held-out data was never used to make a choice, and tuning quietly breaking that rule is the error that makes a model look better than it is.

In this article

Before you start
Why interviewers ask this
Overfitting detection from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page sits in the machine learning algorithms learning path and uses ideas from statistics for machine learning.

Why interviewers ask this

A model that scores perfectly on its own training data can still fail with the first real user, so this question tests whether a candidate evaluates on data the model has not seen. Candidates report it in machine learning interviews. Expect it to lead into follow-ups on regularization, cross-validation and learning curves.

A weak answer says "I look at accuracy" or "I check that the training loss is low." A strong answer names the comparison, the held-out data it needs and the signature of the gap. A Staff-level answer adds what the curves say about the remedy, and the checks, leakage and split mismatch, that decide whether the gap means what it seems to mean.

Overfitting detection from first principles

Validation curve: A validation curve is a plot of the training score and the validation score against the value of one hyperparameter, so the point where the two scores part ways shows where the model starts to overfit.
Learning curve: A learning curve is a plot of the training score and the validation score against the number of training samples, so it shows whether more data would shrink the gap between them.

In plain words

Think of a student who memorizes the answers to last year's exam. On that exact exam the student scores perfectly. On a new exam with different questions the score drops. The student learned the answers, not the subject. A model that overfits behaves the same way.

The analogy breaks in one place. A student knows when they memorized, but a model gives no warning. You must set the new exam yourself, and it must be truly new. If the model has seen it, the test shows nothing.

In one precise sentence, you determine overfitting by measuring one metric on the training data and on held-out data, and a large drop from the first to the second is the sign. Google's Machine Learning Crash Course calls generalization the opposite of overfitting and names two common causes: an overly complex model and unrepresentative training data.

Overfitting shows as a gap between training and held-out scores, and the last check decides whether the gap is real.CC BY 4.0 — source: How do you determine if a model is overfitting?

Step-by-step description

Keep data the model never trains on.
Compare training and validation scores.
A validation curve shows where the gap opens.
A learning curve shows if more data helps.
Check leakage and the split before blaming the model.

Hold out data the model never trains on.
A high training score beside a low validation score means overfitting.
Rule out leakage and a mismatched split before blaming the model.

Step 1: Hold out data the model never sees

Hold out data the model never trains on, because a model's score on its own training data cannot show overfitting. The scikit-learn cross-validation guide calls learning the parameters and testing on the same data a methodological mistake: a model that repeats the labels it has seen would score perfectly and still fail on unseen data.

Cross-validation (CV) is a method that splits the data into k folds, trains on k minus 1 of them, validates on the remaining fold, and repeats until every fold has been the validation fold once. With 30 samples and 10 folds, each round trains on 27 samples and validates on 3.¹ Time-ordered data needs a time-based split, not a shuffled one, because shuffling inflates the validation score.

The trap sits here. Once you pick settings by watching the validation score, that score is biased, as the scikit-learn learning-curve guide warns. Keep a test set untouched until the final check. You now have honest held-out scores to compare.

Step 2: Score both sides and read the gap

A high training score beside a low validation score means overfitting. The scikit-learn guide puts it this way: when both scores are low the estimator is underfitting, and a low training score with a high validation score is usually not possible.

Underfitting is the case where a model is too simple to capture the pattern, so both scores are low. Subtract the validation score from the training score to get the generalization gap.

The scikit-learn polynomial example shows how large the effect can be. It fits 30 noisy samples of a cosine curve and reports 10-fold CV mean squared error (MSE) of 0.408 for degree 1, 0.0432 for degree 4 and 1.83e+08 for degree 15 (scikit-learn 1.9.1 example). The degree-15 error is about 4.2 billion times the degree-4 error, and the degree-1 error is about 9.4 times worse than degree 4.¹ Degree 15 matches the training points almost perfectly but not the true function; degree 1 is too simple.

Step 3: Vary complexity with a validation curve

A validation curve shows where overfitting starts by sweeping one hyperparameter and plotting both scores. A hyperparameter is a setting you choose before training, such as the maximum depth of a decision tree. The scikit-learn validation curve guide shows that a validation curve computes training scores as well as validation scores, which is what makes the gap visible. Sweep the setting over a sensible range and plot both lines on one chart.

In this illustrative depth sweep the validation score peaks at depth 6 while the training score keeps rising, so the gap widens from 0.01 to 0.21.
Tree depth	Training score	Validation score	Gap
1	0.71	0.70	0.01
3	0.83	0.81	0.02
6	0.93	0.85	0.08
12	0.99	0.82	0.17
20	1.00	0.79	0.21

Training score rises to 1.00 at depth 20 while the validation score peaks at 0.85 at depth 6.¹ The gap grows from 0.01 at depth 1 to 0.21 at depth 20.¹ Choose the setting where validation peaks, not where training is highest.

Past the depth where validation peaks, extra complexity only widens the gap between training and validation scores.CC BY 4.0 — source: How do you determine if a model is overfitting?

Text description of the diagram

Training score rises to 1.00 at depth 20 while the validation score peaks at 0.85 at depth 6.
The gap grows from 0.01 at depth 1 to 0.21 at depth 20.

Step 4: Vary data size with a learning curve

A learning curve tells you whether more data would shrink the gap. In the scikit-learn documentation a support vector machine with a training score far above its validation score on small data will most likely improve with more samples. For naive Bayes both scores converge to a low value, so more data will not help much.

Take an illustrative overfitting curve. As samples grow from 100 to 6,400, the training score falls from 1.00 to 0.92 and the validation score rises from 0.68 to 0.86, so the gap narrows from 0.32 to 0.06.¹ More data is the right remedy there. In an illustrative underfitting curve, both scores sit near 0.62 at 6,400 samples and the gap is 0.00.¹ That is a bias problem, so the model needs more capacity or better features, as covered in feature engineering techniques, not more rows.

Step 5: Rule out leakage and a mismatched split

Rule out leakage and a mismatched split before blaming the model, because both distort the gap: leakage can make the gap look smaller than it is, and a mismatched split can make it look larger. Data leakage is the use of information from outside the training data, such as test rows or their statistics, while fitting the model, and it makes held-out scores look better than they are. The scikit-learn pitfalls guide recommends a pipeline so that each step, such as scaling, fits on the training subset and only transforms the test subset.

A gap can also appear with no memorization at all. Google's course lists the conditions for good generalization: examples that are independent and identically distributed (IID), stationary, and drawn from similar distributions in every partition. If the validation data comes from a different population, the score drops for that reason. Distribution shift covers how to detect it. Only after these checks does a large gap point to overfitting.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.