Why did you choose a random forest? Variance, out-of-bag scores, baselines

In this article

1. Before you start
2. Why interviewers ask this
3. Random forest from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Random forests are one of the machine learning algorithms interviewers ask about, and this page leans on the statistics fundamentals behind sampling and variance.

Why interviewers ask this

"Why did you choose it?" tests whether a model was a decision or a habit. After a project description, an interviewer can use this question to see if you compared alternatives, can explain the mechanism, and checked the result with evidence.

A weak answer says "it is accurate and everyone uses it." A strong answer names the problem it solves, which is the high variance of a single tree, says what you compared it against, and states how you measured the result. A Staff-level answer also says when the forest is not the right pick and what would change your mind.

The word "optimal" is the trap. No sourced result says a random forest is best on every dataset, so claim "best among the options I tested, on this metric, on held-out data", and show the evidence.

Random forest from first principles

Out-of-bag error

Out-of-bag (OOB) error is the prediction error measured on each row using only the trees whose bootstrap sample did not contain that row, which gives a built-in test-set estimate without a separate validation split.

In plain words

Ask one expert to guess the weight of an ox and the guess may be far off. Ask a hundred people who each saw it from a different angle, average the guesses, and individual quirks cancel. A random forest is that crowd. The analogy breaks if the crowd all copies one neighbour: errors that are shared do not cancel. That is why each tree gets different rows and features.

In one precise sentence, a random forest averages many deep trees whose errors are only partly correlated, which lowers variance, and bias stays close to that of a deep tree or rises slightly (scikit-learn ensemble guide).

Step 1: Random rows and random features make different trees

A single deep tree fits its training data very closely. The scikit-learn guide says individual decision trees typically show high variance and tend to overfit. A forest injects two sources of randomness: bootstrap sampling of the rows and a random subset of features considered at each split. The guide states their purpose is to decrease the variance of the forest estimator, and that the randomness gives trees with somewhat decoupled prediction errors.

Drawing rows with replacement and averaging the models is called bagging. Input: one training table. Output: many different trees. You now know why the trees differ; next, how to test them.

Step 2: Each tree leaves out about a third of the rows

A bootstrap sample of n rows contains about 63.2% of the distinct rows as n grows, so each tree leaves out about 36.8%. For 1,000 rows the formula gives 0.6323 and a simulation gave 0.6316.¹ Those left-out rows are a free test set for that tree. For a forest of 100 trees, each row is left out of about 36.8 trees on average, and the forest scores the row using only those trees.¹ The error from that process is the out-of-bag error.

In scikit-learn, the option oob_score estimates the generalization score from out-of-bag samples, is only available if bootstrap=True, and is off by default. So the claim "I validated it" can rest on this score, then be confirmed on a held-out set.

Step 3: Averaging cuts variance, down to a floor

For n models that each have variance σ², if they were independent, their average would have variance σ²/n. If the n models have equal variance σ² and equal pairwise correlation ρ, the variance of their average is $\rho {\sigma }^2+\frac{1-\rho }{n}{\sigma }^2$ (Hastie, Tibshirani and Friedman, 2009, section 15.2). Take σ² = 1 (illustrative) and ρ = 0.3: the average of 1 tree has variance 1, of 10 trees 0.37, of 100 trees 0.307 and of 1,000 trees 0.3007.¹ With correlation 0 the variance keeps falling as trees are added.

Two lessons follow. Adding trees helps less and less, because the floor ρσ² stays. And lowering the correlation between trees is what moves the floor, which is the job of the random feature subset. Breiman (2001) states that forest error depends on the strength of the individual trees and the correlation between them.

Text description of the diagram

• With correlation 0.3 the variance of the average is 0.307 at 100 trees and 0.3007 at 1,000 trees.
• With correlation 0 the variance keeps falling as trees are added.

Which setting lowers tree correlation?

In scikit-learn (version 1.9.1, as of October 2026) the knob is max_features. The classifier defaults to "sqrt"; the regressor defaults to 1.0, all features. So a default regressor decorrelates trees only through bootstrapping, and lowering max_features is how you lower ρ. The typical effect is lower correlation between trees but weaker individual trees, because each split sees fewer candidate features. This is Breiman's strength–correlation trade-off: forest error depends on both, so tune max_features on out-of-bag or held-out scores rather than assuming smaller is better. Change one setting at a time and compare each score on the same data, so you can say which change helped.

Step 4: More trees do not overfit, and prediction is an average

Breiman shows the generalization error of a forest converges to a limit as the number of trees grows, so adding trees does not cause overfitting. It does not promise good accuracy: the limit can still be poor if the trees are weak. More trees cost compute and memory. For prediction, the ensemble outputs the averaged prediction of the individual trees, as the scikit-learn guide puts it, and a regressor lists a default of 100 trees.

Breiman also reports that random feature selection gives error rates that compare favourably with AdaBoost and are more robust with respect to noise, and that internal estimates measure variable importance. Treat importances as model-internal estimates, not causal effects.

1 · Sample per tree2 · Grow and test3 · Average4 · ConvergeReplay

Step-by-step description

1. Each tree gets its own rows and features.
2. Trees grow alone; left-out rows test each one.
3. Averaging the trees lowers variance.
4. Generalization error converges to a limit; more trees cost compute, not accuracy.

• Each tree trains on a bootstrap sample of rows and a random feature subset at each split.
• Each tree leaves out about a third of the rows, which gives the out-of-bag error.
• The forest averages the trees, which lowers variance.
• The generalization error of a forest converges to a limit as the number of trees grows.

The Staff-level answer

What makes it Staff-level: it states the problem (variance), the evidence (same folds, a held-out check) and the limit (best among what was tried, not best in general). Spoken structure: problem, mechanism, comparison, evidence, caveat.

The rubric

Three variants of this question

Why a random forest instead of gradient boosting?

A forest trains trees independently and averages them, so it is forgiving and easy to parallelise; boosting adds trees one after another to fix earlier errors and can win with more tuning. The scikit-learn guide calls histogram-based gradient boosting "a competitive alternative". Boosting usually works best with shallow trees, bagging with fully grown ones. Say which you tested.

Why not one decision tree or a linear model?

A single deep tree has high variance and overfits, which the forest averages away. A linear model is easier to explain, so compare against it; the linear regression assumptions show when it fits, and regularisation is its variance control.

What if the interviewer says it was not optimal?

Agree that optimal is relative to the metric and the options tried, then say what you would test next, such as tuning, other features from feature engineering, or a boosted model, and how you would compare using the same split.

Common mistakes

• "More trees will make it overfit." (wrong) Breiman shows error converges to a limit as trees grow. (right) "More trees only cost compute once the variance floor is reached."
• "A random forest reduces bias." (wrong) The documented purpose of the randomness is lower variance. (right) "It reduces variance; bias stays low but can rise slightly."
• "I chose it because it is always best." (wrong) No source supports that. (right) "It was the best of the options I compared on this metric."
• "OOB error needs a separate validation set." (wrong) It uses rows each tree left out and requires bootstrap=True. (right) "OOB gives a built-in estimate; I still confirm on held-out data."
• "Feature importance is the causal effect." (wrong) It is an internal estimate of the fitted model. (right) "Importance shows what the model used, not what causes the outcome." Impurity-based importances in scikit-learn also favour high-cardinality features, as the scikit-learn permutation-importance guide notes, so check them with permutation importance.

Follow-up questions

How do you know the random forest was optimal?

Compare it with named baselines on identical folds with one metric chosen in advance, and check that the out-of-bag score and the held-out score agree. A receiver operating characteristic (ROC) curve helps for classification. "Optimal" means best of what you tried.

Does adding more trees make a random forest overfit?

No. Breiman's result is that the generalization error converges as trees are added, so extra trees mostly cost time and memory.

What is out-of-bag error?

It is the error on each row measured only by the trees that never saw that row. It works because each tree leaves out about a third of the rows, and it needs bootstrap sampling turned on.

When would a random forest be a poor choice?

A tree predicts an average of training targets in a leaf, so a regression forest cannot predict beyond the range it has seen. Test it under distribution shift, and prefer something smaller if you must explain every prediction. Very sparse, high-dimensional data and tight latency or memory budgets also count against a forest.

How is training a forest different from gradient descent?

Trees are grown by choosing splits, not by following a loss gradient, and each tree is independent. The gradient descent page covers the optimisation used by models such as linear ones and neural networks.

Sources

1. 1.11 Ensembles: Gradient boosting, random forests, bagging, voting, stacking — scikit-learn documentation (stable, version 1.9.1; date is the read date), 26 Oct 2026
2. RandomForestClassifier — scikit-learn documentation (version 1.9.1; date is the read date), 26 Oct 2026
3. RandomForestRegressor — scikit-learn documentation (version 1.9.1; date is the read date), 26 Oct 2026
4. 5.2 Permutation feature importance — scikit-learn documentation (stable; date is the read date), 26 Oct 2026
5. Random Forests — Leo Breiman, Machine Learning 45, 5-32 (October 2001), 1 Oct 2001
6. The Elements of Statistical Learning, 2nd ed., section 15.2 — Hastie, Tibshirani and Friedman, Springer (2009), 1 Jan 2009