Cross validation explained: what it is and how k-fold works

In this article

1. Before you start
2. Why interviewers ask this
3. Cross-validation from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the machine learning algorithms learning path.

Why interviewers ask this

A model score means little until you say how it was measured. Candidates report that cross-validation comes up in data science and machine learning rounds as a quick test of whether you can tell a trustworthy number from a flattering one.

A weak answer says "split the data into k parts and average the results." A strong answer explains why one split is noisy, what the choice of k trades off, and where the method breaks. The break that matters most is preprocessing done before the split. A Staff-level answer adds three points: the standard deviation of the fold scores is not a confidence interval, ordered data needs ordered splits, and a separate test set is still kept for the final check.

Cross-validation from first principles

Cross-validation

Cross-validation is a resampling method that estimates how well a whole modeling procedure will perform on unseen data by repeatedly training on part of the training data, scoring on the part left out, and averaging the scores.

Fold

A fold is one of the k roughly equal-sized groups of rows that the training data is divided into; each fold is the held-out part exactly once.

In plain words

Imagine you have a bank of practice questions and want to know if you really understand the subject. Scoring yourself on questions you already memorized proves nothing. The scikit-learn documentation calls testing a model on the data it learned from "a methodological mistake".

So you split the bank into five piles. You study four, take the fifth as an exam, and rotate until each pile has been the exam once. The analogy breaks in one place: a model can be retrained from scratch for each rotation, so no attempt remembers an earlier one.

One split has two problems. The score depends on which rows landed in the test part. A hyperparameter is a setting chosen before training, such as the strength of regularization. If you tune hyperparameters against one test set, knowledge about it "leaks" into the model.

In one precise sentence, cross-validation scores the same procedure on k different train and held-out splits, so every row is used for testing once and for training k minus one times.

Step 1: Split the training data into k folds

Input: a training set of n rows. Output: k folds of nearly equal size. First set the test set aside; the scikit-learn guide says to keep one for the final evaluation, and the validation set is no longer needed when you use CV.

Then cut the rows into k groups. With n = 150 rows and k = 5, each fold has 30 rows, so every round trains on 120 rows and tests on 30.¹ By default cross_validate and cross_val_score use 5 folds, and for a classifier they use StratifiedKFold (scikit-learn documentation). Stratified k-fold is a version of k-fold that keeps the class proportions the same in every fold. The default splitters do not shuffle. Now you know the first move: cut once, then never move a row between folds.

Step 2: Train on k minus one folds, score on the held-out fold

Input: the k folds. Output: k scores. Take fold 1 as the held-out part, train a fresh model on the other folds and score it on fold 1. Then take fold 2 as the held-out part and repeat, until every fold has been held out once.

The documentation puts it as "a model is trained using k-1 of the folds as training data" and tested on the rest. Each fold is the held-out set exactly once, so no row is scored by a model that saw it.

The score can be any metric, such as accuracy or the area under a receiver operating characteristic (ROC) curve; see how to compute a ROC curve in scikit-learn.

Each score comes from a model trained on only k minus one folds, which is less data than you will finally use. This is why the estimate leans slightly pessimistic, a point the trade-offs section returns to. You now have k honest numbers; the next step protects them.

Step 3: Fit every preprocessing step inside the training folds

Preprocessing is fitted on the training folds inside every round of Step 2, not after it: in each round the scaler or feature selector learns from the four training folds, then the model trains, then the score is taken on the held-out fold. Data leakage in cross-validation usually comes from preprocessing, not from the folds. Scaling, imputing missing values and feature selection all learn something from data. If you run them on all rows first, every fold's training step has already seen the held-out rows.

The scikit-learn guide says such transformations should be learned from the training part and then applied to held-out data. Its tool for this is Pipeline, which bundles preprocessing and model so each fold refits both.

The Stanford Statistics (STATS) 202 notes show how large the damage is. The setup has 200 rows and 1000 predictors, with a coin-flip response that no predictor relates to. Selecting the 20 most correlated predictors on all the data and then running 10-fold CV gives a misleadingly low error, 3% in the notes, while the true misclassification rate is roughly 50% because the response is a coin flip. The right way selects the predictors inside each fold's training part. The page's script reproduces the setup with a simple classifier and 20 repeats; the mean errors of the two ways are stored under selection_before_cv_mean_error and selection_inside_cv_mean_error in its output file.¹ Feature work belongs inside the loop; see feature engineering techniques.

Step 4: Average the scores and report the spread

Input: k scores. Output: one number plus its spread. The guide states that "the performance measure reported by k-fold cross-validation is then the average of the values computed in the loop."

Report the spread too, but do not read it as a confidence interval. The folds share most of their training rows, so their errors are correlated. Bengio and Grandvalet (2004) prove that no universal unbiased estimator of the variance of k-fold cross-validation exists. They also show that naive estimators that ignore the overlap grossly underestimate it.

Cross-validation estimates performance; it does not hand you a model. The usual practice is to refit the chosen procedure on all the training data and score it once on the test set.

1 · Split2 · Rotate3 · Fit inside4 · AverageReplay

Step-by-step description

1. Cut 150 rows into 5 folds of 30.
2. A different fold is held out each round.
3. Each round fits, trains and scores.
4. Five scores give a mean of 0.976.

• Each fold is the held-out set exactly once.
• Preprocessing is fitted on the training folds inside every round.

Worked example

The scikit-learn guide runs cross_val_score for a linear support vector classifier on the iris data, which has 150 rows, with 5 folds. The fold accuracies are 0.96, 1.0, 0.96, 0.96 and 1.0.¹ Their mean is 0.976, which the documentation prints as 0.98, and their standard deviation is about 0.0196, printed as 0.02 (scikit-learn guide).¹

A single hold-out split with train_test_split and a test size of 0.4 leaves 90 training rows and 60 test rows and returns one score, 0.96, per the same guide.¹ Five folds return five scores, which show the spread that one split hides.

Text description of the diagram

• The five fold accuracies are 0.96, 1.0, 0.96, 0.96 and 1.0.
• Their mean is 0.976, which the documentation prints as 0.98.

Leave-one-out cross-validation (LOOCV) is the special case where k equals n, so each round holds out one row. It "requires fitting the statistical learning method n times", per the Duke lecture notes on chapter 5 of An Introduction to Statistical Learning. On 1,000 rows that is 1,000 fits against 5.¹

Trade-offs and edge cases

Choosing k trades bias against variance and cost. The Duke notes say practice is to use k of 5 or 10 and that k-fold often gives more accurate test-error estimates than LOOCV. A small k trains on less data, which pushes the estimate pessimistic. LOOCV has little of that bias, but its fitted models are almost identical, so it can have a higher-variance estimate (the argument made in chapter 5 of An Introduction to Statistical Learning; reported results differ). Five or ten folds are the compromise. See bias vs variance for the underlying idea.

Choosing between models has its own rule. The Stanford notes give the one-standard-error rule: "Choose the simplest model whose CV error is no more than one standard error above the model with the lowest CV error." It pairs well with penalties such as L1 vs L2 regularization.

Text description of the diagram

• StratifiedKFold is the default splitter for classifiers.
• TimeSeriesSplit always tests on later samples than it trains on.

Ordinary k-fold assumes rows are independent. StratifiedKFold is the default splitter for classifiers, and KFold is used in all other cases (cross_validate documentation). For ordered data, shuffled folds let the model train on the future and test on the past. TimeSeriesSplit always tests on later samples than it trains on. With 6 samples and 5 splits, fold 0 trains on sample 0 and tests on sample 1, while fold 4 trains on samples 0 to 4 and tests on sample 5 (TimeSeriesSplit documentation).¹

The training sets are supersets of earlier ones, and a gap argument drops samples between train and test. Rows from the same person or group belong in the same fold, so use a group-aware splitter such as GroupKFold or StratifiedGroupKFold. For small data, RepeatedStratifiedKFold repeats the stratified split with different random splits. For the case where tomorrow's data differs from yesterday's, see distribution shift.

Why is the best cross-validation score after tuning too optimistic?

The best cross-validation score after tuning is too optimistic because the same folds both choose the setting and score it. If you try many hyperparameter settings and keep the one with the highest CV score, you partly select for luck on those particular splits, so the top score leans upward. The scikit-learn example on nested versus non-nested cross-validation says the non-nested approach "uses the same data to tune model parameters and evaluate model performance", which leaks information and gives an "overly-optimistic score".

Nested cross-validation is an outer loop that estimates performance around an inner loop that tunes hyperparameters, for example with GridSearchCV, so the tuning never sees the folds that score it. If you have a separate test set, scoring the tuned model once on it does the same job; nesting is for when there is no spare test data.

How to answer it in an interview

The spoken structure: define it, give the loop, say how you pick k, then name leakage and the test set.

Common mistakes

• "I scaled and selected features on the full dataset, then cross-validated." (wrong) That leaks held-out information into every fold. (right) "I put scaling and selection in a pipeline so each fold refits them."
• "Cross-validation replaces the test set." (wrong) The scikit-learn guide says a test set should still be held out for final evaluation. (right) "I tune with cross-validation, then score once on the test set."
• "The standard deviation of the fold scores is my confidence interval." (wrong) The folds overlap, so it understates the true variance. (right) "It is a rough spread, not a valid interval."
• "I shuffled the folds of my time series." (wrong) The model trains on the future. (right) "I used forward-chaining splits."
• "My best grid-search CV score is my expected performance." (wrong) Choosing the best of many settings on the same folds biases that score upward. (right) "I score the tuned procedure on a test set or in an outer loop."

Follow-up questions

What is the difference between k-fold and leave-one-out cross-validation?

Leave-one-out cross-validation is k-fold with k equal to n, so it needs n fits instead of k. It has little bias but can have a higher-variance estimate (reported results differ), and k-fold with 5 or 10 folds is the usual compromise.

How do you choose k in k-fold cross-validation?

Start with 5 or 10. Lower k saves compute but trains on less data and leans pessimistic; higher k costs more and approaches leave-one-out.

Why can't you use ordinary k-fold cross-validation on time series?

Shuffled folds mix past and future rows, so a model can learn from data that comes after the rows it is tested on. Use TimeSeriesSplit, where each test fold comes after its training rows.

What is nested cross-validation?

Nested cross-validation is an outer loop that estimates performance around an inner loop that tunes hyperparameters, so the tuning never sees the folds that score it. Use it when there is no separate test set; otherwise keep a test set for one final, untouched estimate, as the scikit-learn guide says, because cross-validation does not replace it.

Is the standard deviation of the fold scores a confidence interval?

No. Folds share training rows, so their errors are correlated, and naive spread estimates grossly underestimate the variance (Bengio and Grandvalet, 2004).

What is data leakage in cross-validation?

Data leakage is any fitting step that sees the held-out rows, most often scaling or feature selection done before the split. Put such steps inside a pipeline.

Sources

1. 3.1. Cross-validation: evaluating estimator performance — scikit-learn documentation (version 1.9.1, accessed 2026-10-26), 26 Oct 2026
2. TimeSeriesSplit — scikit-learn documentation (version 1.9.1, accessed 2026-10-26), 26 Oct 2026
3. cross_validate — scikit-learn documentation (version 1.9.1, accessed 2026-10-26), 26 Oct 2026
4. No Unbiased Estimator of the Variance of K-Fold Cross-Validation — Journal of Machine Learning Research 5 (Bengio and Grandvalet), 1 Sep 2004
5. K-fold cross-validation, and the wrong way and the right way to do cross-validation — Stanford STATS 202 course notes (accessed 2026-10-26), 26 Oct 2026
6. Resampling methods: cross-validation (lecture notes after An Introduction to Statistical Learning, chapter 5) — Duke University STA 325 (Steorts), 1 Jan 2017
7. Nested versus non-nested cross-validation — scikit-learn documentation (version 1.9.1), 26 Oct 2026