Bagging vs Boosting: How the Two Big Ensemble Ideas Actually Differ
Bagging trains models in parallel to cut variance; boosting trains them in sequence to cut bias. How each works, and when to reach for which.
Ensemble methods rest on a simple observation: a committee of imperfect models, combined well, beats almost any single model. The two foundational ways to build that committee — bagging and boosting — sound similar and are near-opposites. Understanding which problem each attacks (in bias–variance terms) tells you when to use which.
Bagging: parallel voters against variance
Bagging — *bootstrap aggregating* — trains many copies of the same model type on different bootstrap samples (random draws with replacement) of the training data, then averages their predictions (or takes a majority vote). Each individual model overfits its own sample in its own way, but their errors are partly independent, so averaging cancels much of the noise. Bagging is therefore a variance reducer: it works best with deep, unstable, low-bias learners like fully grown decision trees. The random forest takes this further by also randomizing which features each tree split can consider, decorrelating the trees so the averaging works even better.
Boosting: sequential specialists against bias
Boosting trains models one after another, and each new model focuses on what the ensemble so far gets wrong. In gradient boosting, each new small tree is fitted to the current errors (residuals) of the ensemble and added with a small weight — hundreds of shallow, individually weak trees stacking corrections into a very accurate whole. Boosting is a bias reducer: it turns underpowered learners into a powerful one. The trade-off: because it keeps chasing the remaining errors, boosting can eventually chase noise, so it needs careful tuning (learning rate, tree depth, number of rounds, early stopping).
| Bagging | Boosting | |
|---|---|---|
| Training | Parallel, independent models | Sequential — each corrects the last |
| Primarily reduces | Variance | Bias |
| Base learners | Deep, unstable (fully grown trees) | Shallow, weak (stumps or small trees) |
| Overfitting risk | Low — averaging protects you | Higher — needs tuning and early stopping |
| Sensitivity to noisy labels | Robust | Can obsess over mislabeled points |
| Typical example | Random forest | Gradient-boosted trees |
A practical default
On tabular data, a random forest is a superb low-maintenance baseline — hard to badly misconfigure. Well-tuned gradient boosting usually edges ahead on accuracy and remains the method to beat on tabular problems. Reach for the forest first, boost when you need the last few points.
Go deeper on the ensemble methods (bagging, boosting, stacking) topic page — stacking being the third idea, where a meta-model learns to combine diverse base models. And evaluate any ensemble honestly with cross-validation. **AI Learning** includes ensembles throughout its cheat sheets and ~10,000 offline practice questions, from intuition to interview depth.
Free · Works offline · Ensembles, tuning and evaluation — practiced, not just read.
Sources
- Breiman (1996) — 'Bagging Predictors', Machine Learning
- Breiman (2001) — 'Random Forests', Machine Learning
- Friedman (2001) — 'Greedy Function Approximation: A Gradient Boosting Machine', Annals of Statistics
Frequently asked questions
More in Learn AI & Data Science
- Coding Interview Strategy: A Framework for Solving Problems Under PressureKnowing algorithms isn't enough — you have to deploy them under pressure. Here's a repeatable framework for solving unseen problems and a study plan that builds durable skill.
- P vs NP and Complexity Classes: What 'Hard' Really Means (Intuition, No Proofs)Why do some problems have fast algorithms and others resist every attempt? P vs NP is the deepest open question in computer science — here's the intuition, minus the proofs.
- Union-Find (Disjoint Set Union): Near-Constant Connectivity with Two OptimizationsUnion-Find answers 'are these two things connected?' and 'connect them' in almost O(1) — thanks to two beautifully simple optimizations that make the trees nearly flat.