Bagging vs boosting: what is the difference?

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the machine learning algorithms learning path, and the statistics under it sit in probability and statistics.

Why interviewers ask this

A candidate who mixes up bagging and boosting usually has not understood bias and variance, and those two ideas sit under most model-choice decisions. Candidates report being asked to explain voting, bagging and boosting together, and to say why a random forest beats a single tree.

A weak answer says "boosting is better because it is more accurate" or "both just combine models." A strong answer says what each method changes about how the models are built (independent resamples versus a sequence), which error each one attacks (variance versus bias), and what each costs (tuning effort, parallel training, sensitivity to noise). A Staff-level answer adds the limit of averaging, which is the correlation between models, and the way a random forest and stochastic gradient boosting borrow tricks from each other. The sections below build that answer from zero.

What each one is

Bagging

Bagging, short for bootstrap aggregating, is an ensemble method that trains the same kind of model on many bootstrap samples of the training set and combines the outputs by averaging a number or voting on a class. It lowers variance and leaves bias roughly unchanged.

Boosting

Boosting is an ensemble method that adds weak models one at a time, each fitted to the mistakes of the ensemble built so far. The final prediction is the sum of the models' scaled contributions, so the combined model's bias falls round by round.

Bootstrap sample

A bootstrap sample is a training set of the same size as the original, built by drawing rows at random with replacement. Some rows appear several times and others not at all. Each bagged model gets its own bootstrap sample.

Out-of-bag row

An out-of-bag row is a training row that one particular bootstrap sample happened to leave out. That model never saw the row, so the row can be used to test it, which is a common way to estimate error without a separate validation set.

In plain words

Picture two ways to get a good weather forecast. In the first, you ask many forecasters who each studied a different random batch of past records, then average their answers. That is bagging. In the second, one forecaster writes a forecast, you mark where it was wrong, and the next forecaster only corrects those errors. That is boosting.

The analogy breaks in one place. Real bagged models are not truly independent forecasters: they learn from overlapping data and make some of the same mistakes, and that shared part of the error is the correlation term in the worked example below. In one precise sentence, bagging averages independent models to shrink variance, and boosting builds dependent models to shrink bias.

Bagging: independent models on resamples

Bagging trains each model on its own bootstrap sample and aggregates the results. Breiman (1996) forms the versions by making bootstrap replicates of the learning set, with each replicate drawing N cases at random with replacement. The aggregate averages when predicting a number and takes a plurality vote when predicting a class.

The scikit-learn documentation describes the driving principle of averaging methods as building several estimators independently and then averaging their predictions, which usually beats any single estimator because the variance falls. Breiman found gains only for unstable procedures, where a small change in the learning set can cause a large change in the predictor. Deep trees are unstable, so they gain a lot; a stable learner has little variance to remove.

A random forest is bagged decision trees plus a random subset of features considered at each split. The scikit-learn guide says both sources of randomness exist to decrease the variance of the forest, because they make the trees' errors less alike so that some cancel when averaged.

Bagging's left-out rows give a free accuracy estimate

A bootstrap sample leaves out about a third of the training rows, so every bagged model has rows it never saw. The chance that one row is skipped by one sample of n draws is (1 − 1/n)ⁿ, because each of the n draws misses that row with probability 1 − 1/n. For 10 rows it is 0.3487, for 100 rows 0.366, for 1,000 rows 0.3677, and the limit for a large training set is 0.3679.¹

Breiman's retrospective puts it as roughly 37% of the observations not included in the fit, called out-of-bag. A bootstrap sample therefore holds about 63% distinct rows.¹ Scoring each row only with the models that skipped it gives an honest error estimate from the training data alone, with no separate validation set to carve out. You now know how bagging builds and checks its models; boosting does neither.

Boosting: models added in sequence to fix errors

Boosting builds each model after the last. The scikit-learn documentation says boosting builds base estimators sequentially to reduce the bias of the combined estimator, and that the motivation is to combine several weak models into a powerful one. AdaBoost and gradient tree boosting are its examples.

Gradient boosting is boosting in which each new model is fitted to the negative gradient of a loss function. Friedman (2001) connects stagewise additive expansions to steepest-descent minimization. For squared error (written as half the squared error) the negative gradient is the residual, so each round fits a small model to the current residuals. The learning rate is a small number that scales each new model's contribution, so no single round overcorrects. XGBoost is a scalable, regularized implementation of tree boosting; Chen and Guestrin (2016) describe the prediction as the sum of the scores from K trees.

Text description of the diagram

• Bagging trains trees in parallel on bootstrap samples and averages or votes.
• Boosting builds base estimators sequentially to reduce the bias of the combined estimator.

Worked example

Four small computations make the contrast concrete. All inputs are illustrative, and each result is computed rather than quoted.

Voting. Suppose three classifiers are each right 70% of the time (illustrative) and make independent errors. A majority vote is right 0.784 of the time; five such classifiers reach 0.8369.¹ Independence is the catch, and the next paragraph shows what happens without it.

Variance of an average. Take B models whose predictions each have variance 1 (illustrative) and pairwise correlation ρ. The variance of their average is

with σ² = 1. The first term does not shrink with B, so correlated models stop improving.¹

Bootstrap by hand. With six rows numbered 0 to 5, one illustrative draw of 0, 2, 2, 3, 5, 5 uses four distinct rows and leaves rows 1 and 4 out of bag, which is a third of the rows.¹

Boosting a residual. Start from the mean of the targets 2, 4, 6 and 12, which is 6. For a row whose target is 12 the residual is 6. With a learning rate of 0.1 and a weak model that fits that residual exactly (illustrative), the residual is 5.4 after round 1, 4.86 after round 2, 4.374 after round 3 and 2.0921 after round 10.¹

Text description of the diagram

• Averaging 100 independent models of variance one gives variance 0.01.
• At correlation 0.6 the variance of the average of 100 models is still 0.604.

This is why a random forest picks random features at each split: it lowers ρ, the shared part, and so lets averaging do more.

Trade-offs and edge cases

How do bagging and boosting differ side by side?

Bagging and boosting differ in how the models are built, which error they attack and what they cost.

The "independent versus sequential" and "variance versus bias" rows follow the scikit-learn guide. The textbook split is a simplification: a boosted model with many rounds can still overfit, so the learning rate, tree depth, subsampling and regularization control its variance (scikit-learn; Chen and Guestrin, 2016). The regularization idea is the same one covered in L1 vs L2 regularization.

Bagging or boosting: which is better?

Neither wins everywhere; the one that fits depends on where your error sits. Fit one model and compare training and validation error. A large gap means high variance, which points to bagging or a random forest. High error on both means high bias, which points to boosting.

Text description of the diagram

• A large gap between training and validation error points to variance and to bagging.
• High error on both points to bias and to boosting.

Treat this as a heuristic, not a law. A common piece of practical advice is that boosting needs more tuning and is more sensitive to noisy labels, so a small or noisy dataset may favour bagging. That advice comes from practice and secondary guides, not from a primary proof. To compare both on held-out data for a classifier, plot the receiver operating characteristic (ROC) curve of each; see ROC curve in scikit-learn.

Edge cases

Bagging does not help stable learners. Breiman's analysis says the gain depends on how unstable the base procedure is, so bagging a plain least-squares linear model adds compute and little else, although Breiman (1996) reported substantial gains for linear regression with subset selection, which is unstable. The methods also mix. Friedman's stochastic gradient boosting fits each round on a fraction subsample of the data drawn without replacement, and the same page gives 0.5 as a typical value. Finally, a shift in the data after training hurts both; see distribution shift.

How to answer it in an interview

The spoken structure: say how the models are built, say which error each one attacks, give the cost, then give the check you would run first.

Common mistakes

• "Bagging reduces bias." (wrong) Bagging averages models and leaves bias roughly unchanged. (right) "Bagging cuts variance; boosting is what attacks bias."
• "Boosting only reduces bias." (wrong) That is the textbook simplification; many rounds can overfit. (right) "Boosting mainly cuts bias, and I control its variance with learning rate, depth, subsampling and regularization."
• "A random forest is just bagging." (wrong) It omits the random feature subset at each split. (right) "A random forest is bagged trees plus random features per split, which decorrelates the trees."
• "Boosting parallelizes like bagging." (wrong) Each round needs the previous one. (right) "Bagged models train in parallel; boosting rounds are sequential."
• "Boosting always beats bagging." (wrong) It is a trade-off. (right) "Boosting can score higher but costs more tuning."

Follow-up questions

Can boosting overfit, and how do you stop it?

Yes. Every extra round fits the training data more closely, so boosting can overfit. The scikit-learn guide recommends a small learning rate (0.1 or below) and choosing the number of rounds by early stopping, which means halting when the validation error stops improving.

How does bagging reduce variance?

Each model is trained on a different bootstrap sample, so their errors differ, and averaging cancels part of them. The variance of an average of correlated models is $\rho {\sigma }^2+(1-\rho ){\sigma }^2/B$, so the shared part ρσ² is what remains.

How is a random forest different from a decision tree, and why is it better?

A single deep tree has high variance and overfits. A random forest averages many trees built on bootstrap samples with random feature subsets, which decorrelates their errors so the average is steadier (scikit-learn).

What is gradient boosting, and what does XGBoost add?

Gradient boosting fits each new tree to the negative gradient of the loss, which is the residual for squared error. XGBoost adds a regularizer on the trees and systems work for sparse data, cache use and scale (Chen and Guestrin, 2016).

Gini vs entropy: which should a decision tree use?

A decision tree picks each split by trying candidate yes/no questions and keeping the one whose child nodes are least mixed. Gini (1 minus the sum of the squared class shares) and entropy (minus the sum of each class share times its logarithm) both score that mixing and are zero for a pure node, where a class share is the fraction of the node's rows in that class. In practice the two usually choose the same or very similar splits, so the choice rarely matters; Gini is the scikit-learn default and is slightly cheaper because it needs no logarithm.

Why does bagging not reduce bias?

Each bagged model sees data from the same distribution and has the same kind of flexibility, so the average has about the same systematic error as one model. Only the part that varies between training sets averages out.

Sources

1. Ensembles: Gradient boosting, random forests, bagging, voting, stacking (scikit-learn stable) — scikit-learn documentation (living page, last read 2026-10-26), 26 Oct 2026
2. DecisionTreeClassifier (scikit-learn stable reference) — scikit-learn documentation (living page, last read 2026-10-26), 26 Oct 2026
3. Bagging Predictors — Machine Learning 24, 123-140 (Breiman), 1 Aug 1996
4. Greedy function approximation: a gradient boosting machine — Annals of Statistics 29(5) (Friedman), 1 Oct 2001
5. XGBoost: A Scalable Tree Boosting System — ACM KDD 2016 (Chen and Guestrin), 13 Aug 2016
6. Remembering Leo Breiman (arXiv 1101.0917) — arXiv / Annals of Applied Statistics, 1 Jan 2011