Ensemble Learning: Bagging, Boosting, Stacking and LLM Panels

LLM foundationsModels and inferencePublished By Simon Budziak

Ensemble learning is combining several models so their joint prediction beats any single one. The members vote, average, or feed a final model, and the gain comes from their errors differing: when one member is wrong, the others often are not. Random forests, gradient boosting and panels of LLM judges are all ensembles.

How does ensemble learning work?

Ensembles train several base models and merge their outputs. scikit-learn’s guide defines ensemble methods as combining “the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator” (scikit-learn 1.9.1 ensembles guide, read 16 November 2026). Diversity is the whole mechanism: members trained on different samples, features or algorithms make different mistakes, and a vote cancels the ones they do not share. An ensemble of identical models is one model run several times.

What are the main ensemble methods?

Four families cover almost every ensemble in practice.

Method How members are trained How outputs combine Mainly reduces scikit-learn class
Bagging In parallel, each on a bootstrap sample Vote or average Variance BaggingClassifier, RandomForestClassifier
Boosting In sequence, each on the last one’s errors Weighted sum Bias HistGradientBoostingClassifier, AdaBoostClassifier
Stacking Different algorithms on the same data A final model learns how to combine Bias StackingClassifier
Voting Different algorithms on the same data Majority label, or averaged probabilities Variance VotingClassifier

The same guide notes that bagging works best with strong, complex models such as fully grown trees, while boosting usually works best with weak ones such as shallow trees. Stacking trains its final estimator on out-of-sample predictions produced with cross_val_predict, so it does not learn from members’ memorized answers. For tabular data, gradient boosting is the default ensemble to try first; the histogram versions “can be orders of magnitude faster” once samples reach tens of thousands. Outside scikit-learn, XGBoost, LightGBM and CatBoost are the common boosting libraries.

Do ensembles work with large language models?

Yes, mostly at inference time rather than training. Self-consistency samples several chain-of-thought answers from one model and keeps the most common final answer, which the authors report improved GSM8K accuracy by 17.9% (Wang et al., arXiv:2203.11171, ICLR 2023, read 16 November 2026). For grading, a panel of smaller judges from different model families outperformed a single large judge, showed less intra-model bias and was “over seven times less expensive” (Verga et al., arXiv:2404.18796, April 2024, read 16 November 2026). A jury of judges from different providers is an ensemble against judge bias, which makes it a cheap upgrade to LLM-as-a-judge. A mixture of experts is different: one model, trained jointly, routing each token.

When is an ensemble not worth it?

When members share the same blind spot, or when the budget cannot carry it. Five LLM calls cost five times one call and wait on the slowest, so an ensemble belongs on decisions where an error costs more than the extra calls. It also complicates explanation, since no single model owns the answer. Member disagreement is itself a useful signal: high spread is a cheap form of uncertainty estimation, and disagreement is the right trigger for human review, while agreement on a shared mistake still gets through.

This entry was drafted with AI assistance.

Frequently asked questions

Is a random forest an ensemble?

Yes. A random forest is bagging over decision trees, with each split considering only a random subset of features so the trees disagree more and their average is steadier.

Is mixture of experts an ensemble method?

Not in the usual sense. A mixture of experts trains its experts jointly inside one model and a router sends each token to a few of them, while an ensemble combines separate models that each produce a full prediction.

Summarize this page with

Train your team to build this