Ensemble methods (bagging, boosting, stacking)

An ensemble method combines the predictions of multiple individual models (called base learners or weak learners) to produce a single, stronger prediction. The idea is that a group of diverse models, when aggregated, often performs better and more reliably than any single model alone. Common families include bagging, boosting, and stacking.

Bagging stands for Bootstrap Aggregating. It trains many copies of the same base model on different bootstrap samples (random samples drawn with replacement) of the training data, then aggregates their predictions by voting (classification) or averaging (regression). This reduces variance and helps prevent overfitting.

A bootstrap sample is a dataset created by randomly drawing examples from the original training set with replacement, usually of the same size as the original. Because of replacement, some examples appear multiple times while others are left out entirely. Each base model in bagging is trained on its own distinct bootstrap sample to create diversity.

Bagging primarily reduces variance. By averaging the predictions of many models trained on different bootstrap samples, the random fluctuations of individual models cancel out, making the overall prediction more stable. It has relatively little effect on bias, so it works best with low-bias, high-variance models like deep decision trees.

A Random Forest is an ensemble of decision trees built using bagging, with one extra source of randomness: at each split, only a random subset of features is considered. This feature randomness decorrelates the trees so their errors are more independent, which further reduces variance compared with plain bagged trees. Predictions are made by majority vote (classification) or averaging (regression).

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, max_features='sqrt')
model.fit(X_train, y_train)

Boosting builds models sequentially, where each new model focuses on correcting the mistakes made by the previous ones. Models are combined as a weighted sum, with more accurate learners given more influence. This sequential error-correction primarily reduces bias and can turn many weak learners into a single strong learner.

In bagging, base models are trained independently and in parallel on different bootstrap samples, and their results are combined equally. In boosting, base models are trained sequentially, with each model depending on the errors of the previous ones, and combined with different weights. As a result bagging mainly reduces variance while boosting mainly reduces bias.

Stacking combines several different base models by training a higher-level model, called a meta-learner, to learn how to best blend their predictions. The base models make predictions, and these predictions become the input features for the meta-model, which produces the final output. Unlike bagging, stacking usually mixes different types of algorithms.

The meta-learner (also called the level-1 model or blender) takes the predictions of the base models as its input and learns the best way to combine them into a final prediction. Instead of simple averaging, it can learn that some base models are more trustworthy in certain situations. A simple model like logistic regression is often used as the meta-learner.

Different models make different kinds of errors, so when their predictions are combined, individual mistakes tend to cancel out while correct signals reinforce each other. This works best when the models are accurate but also diverse (their errors are not strongly correlated). If all models made the same mistakes, combining them would give no benefit.

A weak learner is a simple model that performs only slightly better than random guessing on its own. Boosting combines many such weak learners sequentially so that together they form a strong learner with high accuracy. A common weak learner is a 'decision stump', a tree with just one split.

AdaBoost assigns weights to training examples and increases the weights of those misclassified by the current model, so the next weak learner pays more attention to the hard cases. Correctly classified points get their weights decreased. Each weak learner also receives an overall weight based on its accuracy when forming the final weighted vote.

from sklearn.ensemble import AdaBoostClassifier
model = AdaBoostClassifier(n_estimators=50, learning_rate=1.0)
model.fit(X_train, y_train)

Gradient Boosting builds models sequentially where each new model is trained to predict the residual errors (the negative gradient of the loss) of the current ensemble. By repeatedly fitting models to these residuals and adding them, the ensemble gradually minimizes the loss function via a form of gradient descent in function space. It is widely used with shallow decision trees as base learners.

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
model.fit(X_train, y_train)

The learning rate scales the contribution of each new tree added to the ensemble, controlling how much each step corrects the residuals. A smaller learning rate makes learning slower and usually requires more trees, but it often improves generalization and reduces overfitting. It is a key hyperparameter that trades off training speed against accuracy.

In hard (majority) voting, each base classifier outputs a class label, and the ensemble predicts the class that receives the most votes. It is the simplest way to aggregate predictions in bagging-style ensembles. Ties may be broken by a rule such as choosing the lowest class index.

Hard voting counts the predicted class labels and picks the majority. Soft voting averages the predicted class probabilities from each model and picks the class with the highest average probability. Soft voting often performs better when the base models output well-calibrated probabilities, because it uses confidence information.

from sklearn.ensemble import VotingClassifier
vc = VotingClassifier(estimators=[('lr',m1),('rf',m2)], voting='soft')
vc.fit(X_train, y_train)

Out-of-bag samples are the training examples that were not selected in a particular bootstrap sample (about 37% of the data on average for each model). These left-out points can be used to evaluate that model's performance, giving an OOB error estimate. This provides a built-in validation score without needing a separate validation set.

On average about 63% of the original unique samples appear in a bootstrap sample, because each draw with replacement gives a probability of (1 - 1/n)^n that a given point is missed, which approaches about 37% for large n. So roughly 37% of points are left out (the out-of-bag samples). This explains where the OOB estimate comes from.

Bagging is most effective with high-variance, low-bias models such as fully grown (deep) decision trees, because these models are unstable and change a lot with small data changes. Averaging many such unstable models cancels out their variance and yields a much more stable predictor. Bagging gives little benefit to already-stable, high-bias models.

Boosting repeatedly increases the focus on hard-to-classify examples, so if those examples are actually mislabeled or noisy, the ensemble may waste capacity fitting that noise. Because each round chases the previous errors, noise can be amplified. Bagging averages independent models, so it is more robust to noisy outliers.

If the meta-learner is trained on base-model predictions made on the same data the base models were trained on, those predictions are overly optimistic and cause data leakage, leading the meta-model to overfit. Using out-of-fold predictions (from cross-validation) ensures the meta-learner sees realistic, unbiased predictions. This keeps the stacked model honest and generalizable.

XGBoost (Extreme Gradient Boosting) is a highly optimized, scalable implementation of gradient-boosted decision trees. It is popular because it includes built-in regularization, efficient handling of missing values, parallelized tree construction, and excellent predictive performance on structured/tabular data. It has won many machine-learning competitions for these reasons.

Feature importance measures how much each feature contributes to the model's predictions, typically by summing how much that feature reduces impurity (or error) across all splits in all trees. Higher importance means the feature is more useful for making accurate splits. Ensembles like random forests and gradient boosting can output these scores to help interpret the model.

import pandas as pd
importances = pd.Series(model.feature_importances_, index=feature_names)
print(importances.sort_values(ascending=False))

The hyperparameter n_estimators sets the number of base trees in the ensemble. In a Random Forest more trees generally improve stability up to a point with diminishing returns, while in gradient boosting more estimators can increase accuracy but also raise the risk of overfitting if not tuned with the learning rate. It directly affects training time and memory use.

A single deep decision tree can memorize the training data, capturing noise and producing high variance. A Random Forest averages many such trees trained on different data and feature subsets, so their individual quirks and noise-fitting cancel out. This averaging dramatically lowers variance while keeping bias roughly the same, improving generalization.

Pasting is an ensemble technique like bagging that trains base models on random subsets of the training data, but it samples without replacement instead of with replacement. Bagging uses bootstrap sampling (with replacement), while pasting draws distinct samples. Both then aggregate predictions by voting or averaging.

Bagging mainly reduces variance by averaging many independent models, leaving bias roughly unchanged. Boosting mainly reduces bias by sequentially fitting the errors of previous models, which can also lower variance but risks higher variance if overdone. Choosing between them depends on whether your base model suffers more from high variance or high bias.

Each weak learner is given a weight (often called alpha) based on its accuracy: more accurate learners receive larger weights and contribute more to the final weighted vote. A learner with high error gets a small or even negative weight, reducing its influence. The final prediction is the sign of the weighted sum of all weak learners' outputs.

Bagging (and methods like Random Forest) allows base models to be trained fully in parallel because each model is independent and uses its own bootstrap sample. Boosting cannot be fully parallelized in the same way because each model depends on the results of the previous one. This makes bagging attractive when fast training on multiple cores is needed.

Level-0 models are the base learners that are trained directly on the original input features and produce predictions. The level-1 model is the meta-learner that takes the level-0 predictions as input and learns how to combine them into the final output. This two-level structure is what gives stacked generalization its name.

Gradient boosting reduces bias by adding many trees sequentially, so each tree only needs to be a weak learner that captures a small piece of the pattern. Shallow trees have high bias but low variance, and stacking many of them gradually builds a strong, low-bias model without overfitting too fast. Using deep trees as base learners would overfit quickly and defeat the boosting strategy.

Ensembles are generally more computationally expensive to train and store because they involve many models, and they are harder to interpret than a single simple model. Prediction can also be slower since every base model must be queried. These costs are the trade-off for the usually higher accuracy and robustness they provide.

If all models are identical in their predictions, combining them gives no improvement over a single model, because there is no diversity for the errors to cancel out. Ensembles only help when base models are both reasonably accurate and make different (uncorrelated) errors. This is why techniques like bootstrap sampling and random feature selection deliberately introduce diversity.

The subsample parameter trains each boosting tree on a random fraction of the training data (less than 100%) rather than all of it. This introduces randomness that decorrelates the trees, reduces variance, and can improve generalization while speeding up training. When subsampling is used, the method is called stochastic gradient boosting.

GradientBoostingClassifier(subsample=0.8, n_estimators=200)