How does ensemble learning work?
Ensembles train several base models and merge their outputs. scikit-learn’s guide defines ensemble methods as combining “the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator” (scikit-learn 1.9.1 ensembles guide, read 16 November 2026). Diversity is the whole mechanism: members trained on different samples, features or algorithms make different mistakes, and a vote cancels the ones they do not share. An ensemble of identical models is one model run several times.
What are the main ensemble methods?
Four families cover almost every ensemble in practice.
| Method | How members are trained | How outputs combine | Mainly reduces | scikit-learn class |
|---|---|---|---|---|
| Bagging | In parallel, each on a bootstrap sample | Vote or average | Variance | BaggingClassifier, RandomForestClassifier |
| Boosting | In sequence, each on the last one’s errors | Weighted sum | Bias | HistGradientBoostingClassifier, AdaBoostClassifier |
| Stacking | Different algorithms on the same data | A final model learns how to combine | Bias | StackingClassifier |
| Voting | Different algorithms on the same data | Majority label, or averaged probabilities | Variance | VotingClassifier |
The same guide notes that bagging works best with strong, complex models such as fully grown trees, while boosting usually works best with weak ones such as shallow trees. Stacking trains its final estimator on out-of-sample predictions produced with cross_val_predict, so it does not learn from members’ memorized answers. For tabular data, gradient boosting is the default ensemble to try first; the histogram versions “can be orders of magnitude faster” once samples reach tens of thousands. Outside scikit-learn, XGBoost, LightGBM and CatBoost are the common boosting libraries.
Do ensembles work with large language models?
Yes, mostly at inference time rather than training. Self-consistency samples several chain-of-thought answers from one model and keeps the most common final answer, which the authors report improved GSM8K accuracy by 17.9% (Wang et al., arXiv:2203.11171, ICLR 2023, read 16 November 2026). For grading, a panel of smaller judges from different model families outperformed a single large judge, showed less intra-model bias and was “over seven times less expensive” (Verga et al., arXiv:2404.18796, April 2024, read 16 November 2026). A jury of judges from different providers is an ensemble against judge bias, which makes it a cheap upgrade to LLM-as-a-judge. A mixture of experts is different: one model, trained jointly, routing each token.
When is an ensemble not worth it?
When members share the same blind spot, or when the budget cannot carry it. Five LLM calls cost five times one call and wait on the slowest, so an ensemble belongs on decisions where an error costs more than the extra calls. It also complicates explanation, since no single model owns the answer. Member disagreement is itself a useful signal: high spread is a cheap form of uncertainty estimation, and disagreement is the right trigger for human review, while agreement on a shared mistake still gets through.