In this article
Before you start
Why interviewers ask this
A wrong diagnosis wastes weeks: a team that adds data to a model that is too simple sees no gain, and a team that adds features to a small-sample regression that already fits its noise makes it worse. Interviewers ask about bias and variance because the answer shows whether a candidate can read two error numbers and choose the next experiment.
A weak answer recites "bias is underfitting, variance is overfitting". A strong answer says how to tell them apart from the training error and the validation error, which fix belongs to which, what the decomposition adds up to, and where the tradeoff stops being true. The question sits in many statistics and machine learning rounds because it takes two minutes and tests judgment, not memory.
What each one is
- Bias
- Bias is the part of a model's prediction error that comes from its assumptions being too simple for the real pattern, measured as the gap between the average prediction over many possible training sets and the true value.
- Variance
- Variance is the part of a model's prediction error that comes from its predictions changing when the training sample changes, measured as the spread of the prediction at one point across many possible training sets.
- Irreducible error
- Irreducible error is the part of a prediction error that comes from noise in the data itself, so that no model, however good, can remove it.
Bias in plain words
Bias is a model's steady, repeatable mistake. Think of an archer whose sight is bent: every arrow lands left of the target, however many are shot. The analogy breaks in one place: an archer sees the target, while a model only sees its training points, so bias is measured against a truth we usually do not know.
The scikit-learn guide (version 1.9.1, read Oct 2026) defines it this way: "The bias of an estimator is its average error for different training sets". It calls a too-simple estimator one that "can at best provide only a poor fit to the samples and the true function because it is too simple (high bias)". The name comes from that steady lean in one direction.
Variance in plain words
Variance is how much a model's answer moves when its training data moves. Think of an archer with a perfect sight and a shaky hand: arrows land around the target but scatter. The analogy breaks in one place: an archer's scatter comes from the hand, while a model's comes from which data it happened to see. A model is random in this sense because its prediction depends on the random sample it was trained on, so predicting at one point is a random variable whose spread is the variance.
The same guide says variance "indicates how sensitive it is to varying training sets". It calls the most flexible estimator one that "approximates the training data perfectly but does not fit the true function very well". High bias shows up as underfitting, a model too simple to fit the pattern. High variance shows up as overfitting, a model so flexible that it fits the noise in one sample.
Why does a degree-9 curve fail while a small penalty rescues it?
A degree-9 polynomial has 10 coefficients, so 30 points leave it enough freedom to chase the noise, and where a sample happens to leave a gap, often near an end of the range, nothing holds the curve down and it swings far from the truth. Its training error of 0.0067 is what least squares with 10 coefficients leaves on 30 points, the noise of 0.01 times (1 minus 10/30).1 Its squared bias was about zero (0.0012 is within the sampling noise of 2,000 sets) and its variance 2.9334, nearly all of its test error of 2.9451.1
Regularization is a penalty on the size of a model's coefficients, so the model cannot bend as far; in scikit-learn's ridge regression the parameter alpha "controls the amount of shrinkage: the larger the value of alpha, the greater the amount of shrinkage" (scikit-learn linear models, version 1.9.1). With a ridge penalty of 0.0001 in our script, variance fell to 0.0021 and squared bias rose from about zero to 0.0002, giving a test error of 0.0123.1
How do the two failures look side by side?
High bias and high variance fail in opposite ways: high bias shows a high training error with a small gap to validation error, and high variance shows a low training error with a large gap.
Text description of the diagram
- High bias shows up as underfitting: the model is too simple, so training error is high and the gap to validation error is small.
- High variance shows up as overfitting: the model is too flexible, so training error is low and the gap to validation error is large.
- High bias needs more capacity or features; high variance needs more data or regularization.
The table summarises the contrast. High bias shows up as underfitting: the model is too simple, so training error is high and the gap to validation error is small. High variance shows up as overfitting: the model is too flexible, so training error is low and the gap to validation error is large. High bias needs more capacity or features; high variance needs more data or regularization.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The setup is a controlled experiment where the truth is known. The true curve is cos(1.5 pi x); each training set is 30 points with noise of standard deviation 0.1, so the irreducible error is 0.01; and 2,000 different training sets were drawn, a polynomial of each degree was fitted to every one, and bias and variance were computed from the 2,000 predictions at 200 test points.1 Real projects have one training set, so they cannot compute either quantity directly; this experiment shows what the two numbers mean.
| Polynomial degree | Squared bias | Variance | Sum with noise 0.01 | Test error | Training error |
|---|---|---|---|---|---|
| 1 | 0.1643 | 0.0165 | 0.1908 | 0.1911 | 0.1749 |
| 2 | 0.0268 | 0.0069 | 0.0437 | 0.0437 | 0.0364 |
| 3 | 0.0024 | 0.0029 | 0.0152 | 0.0152 | 0.0111 |
| 4 | 0.0001 | 0.0023 | 0.0124 | 0.0124 | 0.0085 |
| 6 | 0.0000 | 0.0255 | 0.0355 | 0.0355 | 0.0077 |
| 9 | 0.0012 | 2.9334 | 2.9446 | 2.9451 | 0.0067 |
Read the table in three passes. Squared bias fell from 0.1643 at degree 1 to 0.0268 at degree 2, 0.0024 at degree 3 and about 0.0001 at degree 4.1 Variance was 0.0165 at degree 1, 0.0069 at degree 2, 0.0029 at degree 3, 0.0023 at degree 4 and 0.0255 at degree 6.1 The test error is lowest at degree 4, where it is 0.0124, close to the noise floor of 0.01 that no model can beat. And the training error keeps falling, from 0.1749 to 0.0067, even as the test error at degree 9 rises to 2.9451, which is why training error alone is a poor guide.1
Text description of the diagram
- Squared bias fell from 0.1643 at degree 1 to 0.0268 at degree 2, 0.0024 at degree 3 and about 0.0001 at degree 4.
- Variance was 0.0165 at degree 1, 0.0069 at degree 2, 0.0029 at degree 3, 0.0023 at degree 4 and 0.0255 at degree 6.
Variance also fell from degree 1 to degree 4, which the classical picture does not predict. A line cannot follow the curve, and the part it misses (a squared bias of 0.1643, 16 times the noise of 0.01) behaves like extra noise, so which 30 x-values a sample contains moves the line more than it moves a degree-4 fit.1 "Simple model, low variance" holds when the simple model is roughly right.
Variance is also lumpy. At degree 9 the typical training set (the median of the 2,000) gave a test error of 0.0162, but the worst one gave 2,607.5; at degree 6 the worst was 32.26.1 The average is driven by rare disasters: one of the 2,000 samples supplied 44% of it, and 1 sample in 7 scored worse than 0.1. A single train-and-validate run usually looks fine (median 0.0162) and so hides the risk.1
Now the cure, using the same 2,000 training sets.1
| Penalty strength | Squared bias | Variance | Test error | Training error |
|---|---|---|---|---|
| 0 (none) | 0.0012 | 2.9334 | 2.9459 | 0.0067 |
| 0.000001 | 0.0000 | 0.0039 | 0.0139 | 0.0077 |
| 0.0001 | 0.0002 | 0.0021 | 0.0123 | 0.0084 |
| 0.01 | 0.0107 | 0.0032 | 0.0239 | 0.0187 |
| 1.0 | 0.1606 | 0.0093 | 0.1800 | 0.1638 |
The penalty is a dial between the two failures. At 1.0 the squared bias is 0.1606 and the training error 0.1638, the signature of high bias again, so too much regularization recreates the first problem.1
Trade-offs and edge cases
The decision is always the same: which of the two errors is large, and what lever fits it.
| Question | High bias | High variance |
|---|---|---|
| Training error | high | low |
| Gap to validation error | small | large |
| First fixes | more capacity or features | more data, regularization |
| What more data does | little, the model cannot use it | helps, the sample matters less |
Two edge cases matter. First, more data does not fix high bias. The scikit-learn guide says that when training and validation scores both converge to a low value as the training set grows, "we will probably not benefit much from more training data", while a training score far above the validation score suggests that adding samples "will most likely increase generalization" (scikit-learn guide, version 1.9.1). Second, the tradeoff is not universal.
Belkin et al. (arXiv, 28 December 2018) argue that modern models that fit the training data exactly can still perform well, and propose a "double descent" curve that contains the classical U shape and then improves again as capacity grows past the point of interpolation (Belkin et al., 2018). Neal (arXiv, 17 December 2019) reports that the tradeoff is absent in neural networks when width increases (Neal, 2019). Reported practice differs, so say "classically, with a fixed training set" when you state the tradeoff.
The diagnosis fits in two questions. If training error is high the model has high bias. A model can have both: a training error well above the noise floor and a large gap to validation; fix the bias first, because a bigger model changes the variance too. If training error is low but validation error is much worse the model has high variance. If both errors are low and close the model is balanced.
Text description of the diagram
- If training error is high the model has high bias.
- A model can have both high bias and high variance; fix the bias first.
- If training error is low but validation error is much worse the model has high variance.
- If both errors are low and close the model is balanced.
How to answer it in an interview
What is the difference between bias and variance?
Bias is error from a model being too simple for the real pattern, so it misses the same way on every training set. Variance is error from a model being so flexible that its predictions swing with the training sample. Squared bias plus variance plus the irreducible noise gives the expected test error for squared error. I tell them apart by comparing training and validation error: high training error means bias, and a low training error with a big gap means variance. Bias needs capacity or features, variance needs data or regularization, and I would not assume the tradeoff holds for very large neural networks.
The spoken structure is: define both in one sentence each, say how they add up, give the diagnosis, then the fixes and the caveat.
Common mistakes
"High training accuracy means the model is good." (wrong) A model with a tiny training error can have a huge test error, as the degree-9 run shows. (right) "I judge a model on validation data and compare it with the training error."
"More data always helps." (wrong) If the model is too simple, more samples do not change what it can represent. (right) "More data helps with variance; with high bias I add capacity or better features."
"Bias and variance are the same as underfitting and overfitting." (wrong) They are error components that you can measure in an experiment; underfitting and overfitting are the symptoms you see. (right) "Underfitting is a symptom of high bias, and overfitting a symptom of high variance."
"Regularization always helps." (wrong) Too large a penalty makes the model too simple, with a test error of 0.1800 in the run above. (right) "I choose the penalty on validation data, because it moves the model along the tradeoff."
Follow-up questions
What is bias, and why is it called so in the bias-variance tradeoff?
It is called bias because the model leans consistently in one direction away from the truth, as a bent sight leans every arrow the same way. Formally it is the gap between the average prediction over many training sets and the true value.
What is a random variable, and what is random about it?
A random variable is a quantity whose value depends on chance. Here the prediction at a point is random because it depends on which training sample the model saw, and variance is that prediction's spread across samples. See the statistics guide for the basics.
Which models are most likely to experience this, and why?
Flexible models with few constraints are the usual high-variance cases: high-degree polynomials, decision trees grown without a depth limit, k-nearest neighbours with k = 1, and large networks trained on little data. Rigid models are the usual high-bias cases: a straight line or logistic regression on a curved pattern, naive Bayes (scikit-learn's learning-curve example shows it converging to a low score), and very shallow trees (the assumptions behind the straight-line case are in linear regression assumptions). The cause is the same in each: how far the model can bend compared with how much data pins it down.
Does more data fix high bias?
No. If training error is already high and the gap to validation is small, the model is too simple, and more samples will not change that. Add capacity or better features, and check that the optimizer has converged before blaming the model.
Is the bias-variance tradeoff always true?
Not always: it is the classical picture for a fixed training set, and Belkin et al. and Neal report large neural networks where it does not hold. Regularization is the usual lever for variance, compared in L1 versus L2 regularization. For large language models (LLMs) the same held-out-data habit applies, as the LLM evaluation guide explains.
- Bias is error from a model too simple for the pattern; variance is error from a model too sensitive to the sample.
- Squared bias plus variance plus irreducible noise matches the test error for squared error, to within 0.0005 at every degree of the run.
- High training error means bias; low training error with a large validation gap means variance.
- Regularization trades a little bias for a lot less variance, and too much of it recreates high bias.
- The classical tradeoff is not universal: double descent is reported for modern neural networks.
Sources
- Validation curves: plotting scores to evaluate models (scikit-learn 1.9.1 documentation) — scikit-learn, 24 Oct 2026
- Linear Models: ridge regression (scikit-learn 1.9.1 documentation) — scikit-learn, 24 Oct 2026
- Reconciling modern machine learning practice and the bias-variance trade-off — arXiv (Belkin, Hsu, Ma, Mandal), 28 Dec 2018
- On the Bias-Variance Tradeoff: Textbooks Need an Update — arXiv (Neal), 17 Dec 2019
