How does bagging work?
Three steps. Draw a bootstrap sample: pick training examples at random, with replacement, until the sample is as large as the original set, so some examples appear twice and others not at all. Train one model on that sample. Repeat many times, then combine: majority vote for classification, average for regression. Leo Breiman introduced the method in a 1994 Berkeley technical report, published in the journal Machine Learning in 1996, and showed it helps most when small changes in the data cause large changes in the model (Breiman, Bagging Predictors, read 9 November 2026). Bagging fixes instability, not ignorance: averaging many models that all miss the same pattern still misses it.
Bagging vs boosting: what’s the difference?
| Bagging | Boosting | |
|---|---|---|
| Models trained | In parallel, independently | In sequence, each on the last one’s errors |
| Main effect | Reduces variance | Reduces bias |
| Best base model | Unstable, high-variance (deep trees) | Weak, simple (shallow trees) |
| Risk | Higher compute cost | Overfits noisy labels |
| Famous example | Random forest | Gradient boosting |
Use bagging when your model swings with the data, and boosting when it is too simple to fit it. Both are forms of ensemble learning.
When should you use bagging?
When a single model overfits, meaning it scores well on training data and poorly on new data, and the model type is one that changes a lot with small data changes, such as a deep decision tree. Bagging also gives a free accuracy estimate: the examples left out of each bootstrap sample, the out-of-bag set, act as a built-in test set. It parallelizes perfectly, so it costs compute, not time. It rarely helps stable models like linear regression, where regularization is the cheaper fix.
How do you use bagging in practice?
In Python, scikit-learn’s BaggingClassifier wraps any base model; in version 1.9.1 it defaults to 10 estimators, each trained on a bootstrap sample the size of the training set (scikit-learn docs, read 9 November 2026). In practice most teams reach for a random forest instead, which is bagging over trees with random feature selection added, and tune the number of trees until the out-of-bag score stops improving.