Get Started

Ensemble Methods

A group of weak learners, combined correctly, becomes a strong learner. This is not a metaphor. It is a theorem.

120 minutesBuildPythonFree lesson

Learning Objectives

  • Implement AdaBoost and gradient boosting from scratch and explain how boosting sequentially reduces bias
  • Build a bagging ensemble and demonstrate how averaging decorrelated models reduces variance without increasing bias
  • Compare bagging, boosting, and stacking in terms of what error component each method targets
  • Evaluate ensemble diversity and explain why majority voting accuracy improves with more independent weak learners

The Problem

A single decision tree is fast to train and easy to interpret, but it overfits. A single linear model underfits on complex boundaries. You could spend days engineering the perfect model architecture. Or you could combine a bunch of imperfect models and get something better than any of them individually.

Ensemble methods do exactly this. They are the most reliable technique for winning Kaggle competitions on tabular data, they power most production ML systems, and they illustrate the bias-variance tradeoff in action. Bagging reduces variance. Boosting reduces bias. Stacking learns which models to trust on which inputs.

The Concept

Why Ensembles Work

Suppose you have $N$ independent classifiers, each with accuracy $p > 0.5$. The majority vote has accuracy:

For 21 classifiers each with 60% accuracy, majority vote accuracy is about 74%. With 101 classifiers, it rises to 84%. The errors cancel out when the models make different mistakes.

The key requirement is diversity. If all models make the same errors, combining them helps nothing. Ensembles work because they produce diverse models through:

  • Different training subsets (bagging)
  • Different feature subsets (random forests)
  • Sequential error correction (boosting)
  • Different model families (stacking)

Bagging (Bootstrap Aggregating)

Bagging creates diversity by training each model on a different bootstrap sample of the training data.

A bootstrap sample is drawn with replacement from the original data, same size as the original. About 63.2% of unique samples appear in each bootstrap. The remaining 36.8% (out-of-bag samples) provide a free validation set.

Bagging reduces variance without increasing bias much. Each individual tree overfits to its bootstrap sample, but the overfitting is different for each tree, so averaging cancels out the noise.

Random Forests are bagging with an extra twist: at each split, only a random subset of features is considered. This forces even more diversity among trees. The typical number of candidate features is sqrt(n_features) for classification and n_features / 3 for regression.

Boosting (Sequential Error Correction)

Boosting trains models sequentially. Each new model focuses on the examples that previous models got wrong.

Boosting reduces bias. Each new model corrects the systematic errors of the ensemble so far. The final prediction is a weighted sum of all models, where better models get higher weights.

The tradeoff: boosting can overfit if you run too many rounds, because it keeps fitting harder examples, some of which may be noise.

AdaBoost

AdaBoost (Adaptive Boosting) was the first practical boosting algorithm. It works with any base learner, typically decision stumps (depth-1 trees).

The algorithm:

Code
1. Initialize sample weights: w_i = 1/N for all i

2. For t = 1 to T:
   a. Train weak learner h_t on weighted data
   b. Compute weighted error:
      err_t = sum(w_i * I(h_t(x_i) != y_i)) / sum(w_i)
   c. Compute model weight:
      alpha_t = 0.5 * ln((1 - err_t) / err_t)
   d. Update sample weights:
      w_i = w_i * exp(-alpha_t * y_i * h_t(x_i))
   e. Normalize weights to sum to 1

3. Final prediction: H(x) = sign(sum(alpha_t * h_t(x)))

Models with lower error get higher alpha. Misclassified samples get higher weights so the next model focuses on them.

Gradient Boosting

Gradient boosting generalizes boosting to arbitrary loss functions. Instead of reweighting samples, it fits each new model to the residuals (negative gradient of the loss) of the current ensemble.

Code
1. Initialize: F_0(x) = argmin_c sum(L(y_i, c))

2. For t = 1 to T:
   a. Compute pseudo-residuals:
      r_i = -dL(y_i, F_{t-1}(x_i)) / dF_{t-1}(x_i)
   b. Fit a tree h_t to the residuals r_i
   c. Find optimal step size:
      gamma_t = argmin_gamma sum(L(y_i, F_{t-1}(x_i) + gamma * h_t(x_i)))
   d. Update:
      F_t(x) = F_{t-1}(x) + learning_rate * gamma_t * h_t(x)

3. Final prediction: F_T(x)

For squared error loss, the pseudo-residuals are just the actual residuals: $r_i = y_i - F_{t-1}(x_i)$. Each tree literally fits the errors of the previous ensemble.

The learning rate (shrinkage) controls how much each tree contributes. Smaller learning rates require more trees but generalize better. Typical values: 0.01 to 0.3.

XGBoost: Why It Dominates Tabular Data

XGBoost (eXtreme Gradient Boosting) is gradient boosting with engineering optimizations that make it fast, accurate, and resistant to overfitting:

  • Regularized objective: L1 and L2 penalties on leaf weights prevent individual trees from being too confident
  • Second-order approximation: Uses both first and second derivatives of the loss, giving better split decisions
  • Sparsity-aware splits: Handles missing values natively by learning the best direction for missing data at each split
  • Column subsampling: Like random forests, samples features at each split for diversity
  • Weighted quantile sketch: Efficiently finds split points for continuous features on distributed data
  • Cache-aware block structure: Memory layout optimized for CPU cache lines

For tabular data, XGBoost (and its successor LightGBM) consistently outperforms neural networks. This is not changing anytime soon. If your data fits in a table with rows and columns, start with gradient boosting.

Stacking (Meta-Learning)

Stacking uses the predictions of multiple base models as features for a meta-learner.

The meta-learner learns which base model to trust for which inputs. If the random forest is better at certain regions and the SVM at others, the meta-learner will learn to route accordingly.

To avoid data leakage, base model predictions must be generated via cross-validation on the training set. You never train base models and generate meta-features on the same data.

Voting

The simplest ensemble. Just combine predictions directly.

  • Hard voting: Majority vote on class labels.
  • Soft voting: Average predicted probabilities, pick the class with highest average probability. Usually better because it uses confidence information.

Build It

Step 1: Decision Stump (Base Learner)

The code in code/ensembles.py implements everything from scratch. We start with a decision stump: a tree with a single split.

Python
class DecisionStump:
    def __init__(self):
        self.feature_idx = None
        self.threshold = None
        self.polarity = 1
        self.alpha = None

    def fit(self, X, y, weights):
        n_samples, n_features = X.shape
        best_error = float("inf")

        for f in range(n_features):
            thresholds = np.unique(X[:, f])
            for thresh in thresholds:
                for polarity in [1, -1]:
                    pred = np.ones(n_samples)
                    pred[polarity * X[:, f] < polarity * thresh] = -1
                    error = np.sum(weights[pred != y])
                    if error < best_error:
                        best_error = error
                        self.feature_idx = f
                        self.threshold = thresh
                        self.polarity = polarity

    def predict(self, X):
        n = X.shape[0]
        pred = np.ones(n)
        idx = self.polarity * X[:, self.feature_idx] < self.polarity * self.threshold
        pred[idx] = -1
        return pred

Step 2: AdaBoost from Scratch

Python
class AdaBoostScratch:
    def __init__(self, n_estimators=50):
        self.n_estimators = n_estimators
        self.stumps = []
        self.alphas = []

    def fit(self, X, y):
        n = X.shape[0]
        weights = np.full(n, 1 / n)

        for _ in range(self.n_estimators):
            stump = DecisionStump()
            stump.fit(X, y, weights)
            pred = stump.predict(X)

            err = np.sum(weights[pred != y])
            err = np.clip(err, 1e-10, 1 - 1e-10)

            alpha = 0.5 * np.log((1 - err) / err)
            weights *= np.exp(-alpha * y * pred)
            weights /= weights.sum()

            stump.alpha = alpha
            self.stumps.append(stump)
            self.alphas.append(alpha)

    def predict(self, X):
        total = sum(a * s.predict(X) for a, s in zip(self.alphas, self.stumps))
        return np.sign(total)

Step 3: Gradient Boosting from Scratch

Python
class GradientBoostingScratch:
    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):
        self.n_estimators = n_estimators
        self.lr = learning_rate
        self.max_depth = max_depth
        self.trees = []
        self.initial_pred = None

    def fit(self, X, y):
        self.initial_pred = np.mean(y)
        current_pred = np.full(len(y), self.initial_pred)

        for _ in range(self.n_estimators):
            residuals = y - current_pred
            tree = SimpleRegressionTree(max_depth=self.max_depth)
            tree.fit(X, residuals)
            update = tree.predict(X)
            current_pred += self.lr * update
            self.trees.append(tree)

    def predict(self, X):
        pred = np.full(X.shape[0], self.initial_pred)
        for tree in self.trees:
            pred += self.lr * tree.predict(X)
        return pred

Step 4: Compare against sklearn

The code verifies that our from-scratch implementations produce similar accuracy to sklearn's AdaBoostClassifier and GradientBoostingClassifierand compares all methods side by side.

Use It

When to Use Each Method

MethodReducesBest forWatch out for
Bagging / Random ForestVarianceNoisy data, many featuresDoes not help with bias
AdaBoostBiasClean data, simple base learnersSensitive to outliers and noise
Gradient BoostingBiasTabular data, competitionsSlow to train, easy to overfit without tuning
XGBoost / LightGBMBothProduction tabular MLMany hyperparameters
StackingBothGetting last 1-2% accuracyComplex, risk of overfitting meta-learner
VotingVarianceQuick combination of diverse modelsOnly helps if models are diverse

The Production Stack for Tabular Data

For most tabular prediction problems, this is the order to try:

  1. LightGBM or XGBoost with default parameters
  2. Tune n_estimators, learning_rate, max_depth, min_child_weight
  3. If you need the last 0.5%, build a stacking ensemble with 3-5 diverse models
  4. Use cross-validation throughout

Neural networks on tabular data are almost always worse than gradient boosting, despite continued research attempts. TabNet, NODE, and similar architectures occasionally match but rarely beat a well-tuned XGBoost.

Ship It

This lesson produces outputs/prompt-ensemble-selector.md -- a prompt that helps you pick the right ensemble method for a given dataset. Describe your data (size, feature types, noise level, class balance) and the problem you are solving. The prompt walks through a decision checklist, recommends a method, suggests starting hyperparameters, and warns about common mistakes for that method. Also produces outputs/skill-ensemble-builder.md with the full selection guide.

Exercises

  1. Modify the AdaBoost implementation to track training accuracy after each round. Plot accuracy vs. number of estimators. When does it converge?
  2. Implement a random forest from scratch by adding random feature subsampling to the regression tree. Train 100 trees with max_features=sqrt(n_features) and average predictions. Compare variance reduction to a single tree.
  3. In the gradient boosting implementation, add early stopping: track validation loss after each round and stop when it has not improved for 10 consecutive rounds. How many trees does it actually need?
  4. Build a stacking ensemble with three base models (logistic regression, decision tree, k-nearest neighbors) and a logistic regression meta-learner. Use 5-fold cross-validation to generate meta-features. Compare to each base model alone.
  5. Run XGBoost on the same dataset with default parameters. Compare its accuracy to your from-scratch gradient boosting. Time both. How large is the speed difference?

Key Terms

TermWhat people sayWhat it actually means
Bagging"Train on random subsets"Bootstrap aggregating: train models on bootstrap samples, average predictions to reduce variance
Boosting"Focus on hard examples"Train models sequentially, each correcting errors of the ensemble so far, to reduce bias
AdaBoost"Reweight the data"Boosting via sample weight updates; misclassified points get higher weight for the next learner
Gradient boosting"Fit the residuals"Boosting via fitting each new model to the negative gradient of the loss function
XGBoost"The Kaggle weapon"Gradient boosting with regularization, second-order optimization, and systems-level speed tricks
Stacking"Models on top of models"Use predictions of base models as input features for a meta-learner
Random forest"Many randomized trees"Bagging with decision trees, adding random feature subsampling at each split for diversity
Ensemble diversity"Make different mistakes"Models must be uncorrelated in their errors for the ensemble to improve over individuals
Out-of-bag error"Free validation"Samples not in a bootstrap draw (~36.8%) serve as a validation set without needing a holdout

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

ensembles.py
import numpy as np
from collections import Counter


def make_classification_data(n_samples=300, n_features=5, noise=0.1, seed=42):
    rng = np.random.RandomState(seed)
    X = rng.randn(n_samples, n_features)
    boundary = 0.5 * X[:, 0] + 0.3 * X[:, 1] ** 2 - 0.2 * X[:, 2]
    y = np.where(boundary + rng.normal(0, noise, n_samples) > 0, 1, -1)
    return X, y


def make_regression_data(n_samples=300, n_features=5, noise=0.3, seed=42):
    rng = np.random.RandomState(seed)
    X = rng.randn(n_samples, n_features)
    y = 2.0 * X[:, 0] + np.sin(3 * X[:, 1]) - 0.5 * X[:, 2] ** 2 + rng.normal(0, noise, n_samples)
    return X, y


def train_test_split(X, y, test_ratio=0.2, seed=42):
    rng = np.random.RandomState(seed)
    idx = rng.permutation(len(y))
    split = int(len(y) * (1 - test_ratio))
    return X[idx[:split]], X[idx[split:]], y[idx[:split]], y[idx[split:]]


class DecisionStump:
    def __init__(self):
        self.feature_idx = None
        self.threshold = None
        self.polarity = 1
        self.alpha = None

    def fit(self, X, y, weights):
        n_samples, n_features = X.shape
        best_error = float("inf")

        for f in range(n_features):
            thresholds = np.unique(X[:, f])
            for thresh in thresholds:
                for polarity in [1, -1]:
                    pred = np.ones(n_samples)
                    pred[polarity * X[:, f] < polarity * thresh] = -1
                    error = np.sum(weights[pred != y])
                    if error < best_error:
                        best_error = error
                        self.feature_idx = f
                        self.threshold = thresh
                        self.polarity = polarity

    def predict(self, X):
        n = X.shape[0]
        pred = np.ones(n)
        idx = self.polarity * X[:, self.feature_idx] < self.polarity * self.threshold
        pred[idx] = -1
        return pred


class AdaBoostScratch:
    def __init__(self, n_estimators=50):
        self.n_estimators = n_estimators
        self.stumps = []
        self.alphas = []

    def fit(self, X, y):
        n = X.shape[0]
        weights = np.full(n, 1 / n)

        for t in range(self.n_estimators):
            stump = DecisionStump()
            stump.fit(X, y, weights)
            pred = stump.predict(X)

            err = np.sum(weights[pred != y])
            err = np.clip(err, 1e-10, 1 - 1e-10)

            alpha = 0.5 * np.log((1 - err) / err)
            weights *= np.exp(-alpha * y * pred)
            weights /= weights.sum()

            stump.alpha = alpha
            self.stumps.append(stump)
            self.alphas.append(alpha)

    def predict(self, X):
        total = sum(a * s.predict(X) for a, s in zip(self.alphas, self.stumps))
        return np.sign(total)

    def accuracy(self, X, y):
        return np.mean(self.predict(X) == y)


class TreeNode:
    def __init__(self, value=None):
        self.feature_idx = None
        self.threshold = None
        self.left = None
        self.right = None
        self.value = value


class SimpleRegressionTree:
    def __init__(self, max_depth=3, min_samples_split=2):
        self.max_depth = max_depth
        self.min_samples_split = min_samples_split
        self.root = None

    def fit(self, X, y):
        self.root = self._build(X, y, depth=0)

    def _build(self, X, y, depth):
        n_samples, n_features = X.shape

        if depth >= self.max_depth or n_samples < self.min_samples_split:
            return TreeNode(value=np.mean(y))

        best_gain = -float("inf")
        best_feature = None
        best_threshold = None

        current_var = np.var(y) * n_samples

        for f in range(n_features):
            thresholds = np.unique(X[:, f])
            if len(thresholds) > 20:
                thresholds = np.percentile(X[:, f], np.linspace(0, 100, 20))

            for thresh in thresholds:
                left_mask = X[:, f] <= thresh
                right_mask = ~left_mask

                if left_mask.sum() < 1 or right_mask.sum() < 1:
                    continue

                left_var = np.var(y[left_mask]) * left_mask.sum()
                right_var = np.var(y[right_mask]) * right_mask.sum()
                gain = current_var - left_var - right_var

                if gain > best_gain:
                    best_gain = gain
                    best_feature = f
                    best_threshold = thresh

        if best_feature is None or best_gain <= 0:
            return TreeNode(value=np.mean(y))

        left_mask = X[:, best_feature] <= best_threshold
        node = TreeNode()
        node.feature_idx = best_feature
        node.threshold = best_threshold
        node.left = self._build(X[left_mask], y[left_mask], depth + 1)
        node.right = self._build(X[~left_mask], y[~left_mask], depth + 1)
        return node

    def predict(self, X):
        return np.array([self._predict_one(x, self.root) for x in X])

    def _predict_one(self, x, node):
        if node.value is not None:
            return node.value
        if x[node.feature_idx] <= node.threshold:
            return self._predict_one(x, node.left)
        return self._predict_one(x, node.right)


class GradientBoostingScratch:
    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):
        self.n_estimators = n_estimators
        self.lr = learning_rate
        self.max_depth = max_depth
        self.trees = []
        self.initial_pred = None

    def fit(self, X, y):
        self.initial_pred = np.mean(y)
        current_pred = np.full(len(y), self.initial_pred)

        for _ in range(self.n_estimators):
            residuals = y - current_pred
            tree = SimpleRegressionTree(max_depth=self.max_depth)
            tree.fit(X, residuals)
            update = tree.predict(X)
            current_pred += self.lr * update
            self.trees.append(tree)

    def predict(self, X):
        pred = np.full(X.shape[0], self.initial_pred)
        for tree in self.trees:
            pred += self.lr * tree.predict(X)
        return pred

    def mse(self, X, y):
        return np.mean((self.predict(X) - y) ** 2)


class BaggingClassifier:
    def __init__(self, n_estimators=20, max_depth=5):
        self.n_estimators = n_estimators
        self.max_depth = max_depth
        self.trees = []

    def fit(self, X, y):
        rng = np.random.RandomState(42)
        n = len(y)

        for _ in range(self.n_estimators):
            idx = rng.choice(n, size=n, replace=True)
            tree = SimpleRegressionTree(max_depth=self.max_depth)
            tree.fit(X[idx], y[idx])
            self.trees.append(tree)

    def predict(self, X):
        predictions = np.array([tree.predict(X) for tree in self.trees])
        return np.sign(np.mean(predictions, axis=0))

    def accuracy(self, X, y):
        return np.mean(self.predict(X) == y)


class StackingClassifier:
    def __init__(self, base_models, meta_lr=0.1, n_folds=5):
        self.base_models = base_models
        self.meta_lr = meta_lr
        self.n_folds = n_folds
        self.meta_weights = None
        self.meta_bias = None
        self.fitted_models = []

    def fit(self, X, y):
        n = len(y)
        meta_features = np.zeros((n, len(self.base_models)))

        fold_size = n // self.n_folds
        indices = np.arange(n)

        for fold in range(self.n_folds):
            val_start = fold * fold_size
            val_end = val_start + fold_size if fold < self.n_folds - 1 else n
            val_idx = indices[val_start:val_end]
            train_idx = np.concatenate([indices[:val_start], indices[val_end:]])

            for m_idx, model_class in enumerate(self.base_models):
                model = model_class()
                model.fit(X[train_idx], y[train_idx])
                meta_features[val_idx, m_idx] = model.predict(X[val_idx])

        self.meta_weights = np.zeros(len(self.base_models))
        self.meta_bias = 0.0

        for _ in range(200):
            logits = meta_features @ self.meta_weights + self.meta_bias
            preds = np.tanh(logits)
            errors = y - preds
            grad_w = -2 * meta_features.T @ errors / n
            grad_b = -2 * np.sum(errors) / n
            self.meta_weights -= self.meta_lr * grad_w
            self.meta_bias -= self.meta_lr * grad_b

        self.fitted_models = []
        for model_class in self.base_models:
            model = model_class()
            model.fit(X, y)
            self.fitted_models.append(model)

    def predict(self, X):
        meta_features = np.column_stack([m.predict(X) for m in self.fitted_models])
        logits = meta_features @ self.meta_weights + self.meta_bias
        return np.sign(logits)

    def accuracy(self, X, y):
        return np.mean(self.predict(X) == y)


def demo_adaboost():
    print("=" * 60)
    print("ADABOOST FROM SCRATCH")
    print("=" * 60)

    X, y = make_classification_data(n_samples=400, n_features=5)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    for n_est in [1, 5, 10, 25, 50]:
        model = AdaBoostScratch(n_estimators=n_est)
        model.fit(X_train, y_train)
        train_acc = model.accuracy(X_train, y_train)
        test_acc = model.accuracy(X_test, y_test)
        print(f"  n_estimators={n_est:>3d}  train_acc={train_acc:.3f}  test_acc={test_acc:.3f}")

    print()

    stump = DecisionStump()
    stump.fit(X_train, y_train, np.full(len(y_train), 1 / len(y_train)))
    stump_acc = np.mean(stump.predict(X_test) == y_test)
    print(f"  Single stump accuracy: {stump_acc:.3f}")
    print(f"  AdaBoost 50 accuracy:  {model.accuracy(X_test, y_test):.3f}")
    print(f"  Improvement: {model.accuracy(X_test, y_test) - stump_acc:.3f}")
    print()


def demo_gradient_boosting():
    print("=" * 60)
    print("GRADIENT BOOSTING FROM SCRATCH")
    print("=" * 60)

    X, y = make_regression_data(n_samples=400, n_features=5)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    for n_est in [1, 10, 50, 100, 200]:
        model = GradientBoostingScratch(n_estimators=n_est, learning_rate=0.1)
        model.fit(X_train, y_train)
        train_mse = model.mse(X_train, y_train)
        test_mse = model.mse(X_test, y_test)
        print(f"  n_estimators={n_est:>3d}  train_mse={train_mse:.4f}  test_mse={test_mse:.4f}")

    print()

    single_tree = SimpleRegressionTree(max_depth=3)
    single_tree.fit(X_train, y_train)
    tree_mse = np.mean((single_tree.predict(X_test) - y_test) ** 2)
    print(f"  Single tree MSE: {tree_mse:.4f}")
    print(f"  GBM 100 MSE:     {model.mse(X_test, y_test):.4f}")
    print()


def demo_learning_rate_effect():
    print("=" * 60)
    print("LEARNING RATE vs NUMBER OF TREES")
    print("=" * 60)

    X, y = make_regression_data(n_samples=400)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    configs = [
        (0.5, 20),
        (0.1, 100),
        (0.05, 200),
        (0.01, 500),
    ]

    for lr, n_est in configs:
        model = GradientBoostingScratch(n_estimators=n_est, learning_rate=lr)
        model.fit(X_train, y_train)
        test_mse = model.mse(X_test, y_test)
        print(f"  lr={lr:.2f}, n_trees={n_est:>3d}  test_mse={test_mse:.4f}")

    print()
    print("Lower learning rates need more trees but often generalize better.")
    print()


def demo_bagging():
    print("=" * 60)
    print("BAGGING CLASSIFIER")
    print("=" * 60)

    X, y = make_classification_data(n_samples=400)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    single_tree = SimpleRegressionTree(max_depth=5)
    single_tree.fit(X_train, y_train)
    single_acc = np.mean(np.sign(single_tree.predict(X_test)) == y_test)

    bagging = BaggingClassifier(n_estimators=20, max_depth=5)
    bagging.fit(X_train, y_train)
    bag_acc = bagging.accuracy(X_test, y_test)

    print(f"  Single tree accuracy: {single_acc:.3f}")
    print(f"  Bagging (20 trees):   {bag_acc:.3f}")
    print(f"  Variance reduction:   {bag_acc - single_acc:+.3f}")
    print()


def demo_stacking():
    print("=" * 60)
    print("STACKING ENSEMBLE")
    print("=" * 60)

    X, y = make_classification_data(n_samples=400)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    def make_tree_d3():
        return SimpleRegressionTree(max_depth=3)

    def make_tree_d5():
        return SimpleRegressionTree(max_depth=5)

    def make_tree_d7():
        return SimpleRegressionTree(max_depth=7)

    make_tree_d3.fit = None
    make_tree_d5.fit = None
    make_tree_d7.fit = None

    class TreeWrapper:
        def __init__(self, max_depth):
            self.max_depth = max_depth
            self.tree = None

        def fit(self, X, y):
            self.tree = SimpleRegressionTree(max_depth=self.max_depth)
            self.tree.fit(X, y)

        def predict(self, X):
            return np.sign(self.tree.predict(X))

    base_models = [
        lambda: TreeWrapper(3),
        lambda: TreeWrapper(5),
        lambda: TreeWrapper(7),
    ]

    stack = StackingClassifier(base_models=base_models, meta_lr=0.05)
    stack.fit(X_train, y_train)

    for depth, model_fn in zip([3, 5, 7], base_models):
        m = model_fn()
        m.fit(X_train, y_train)
        acc = np.mean(m.predict(X_test) == y_test)
        print(f"  Tree depth={depth} accuracy: {acc:.3f}")

    stack_acc = stack.accuracy(X_test, y_test)
    print(f"  Stacking accuracy:    {stack_acc:.3f}")
    print(f"  Meta-learner weights: {stack.meta_weights}")
    print()


def demo_comparison():
    print("=" * 60)
    print("FULL COMPARISON")
    print("=" * 60)

    X, y = make_classification_data(n_samples=500)
    X_train, X_test, y_train, y_test = train_test_split(X, y)

    single = SimpleRegressionTree(max_depth=5)
    single.fit(X_train, y_train)
    print(f"  Single tree (d=5):    {np.mean(np.sign(single.predict(X_test)) == y_test):.3f}")

    bag = BaggingClassifier(n_estimators=20, max_depth=5)
    bag.fit(X_train, y_train)
    print(f"  Bagging (20, d=5):    {bag.accuracy(X_test, y_test):.3f}")

    ada = AdaBoostScratch(n_estimators=50)
    ada.fit(X_train, y_train)
    print(f"  AdaBoost (50 stumps): {ada.accuracy(X_test, y_test):.3f}")

    print()
    print("Bagging reduces variance (better than single tree).")
    print("Boosting reduces bias (learns complex boundaries from weak learners).")
    print()


def demo_sklearn_comparison():
    print("=" * 60)
    print("SKLEARN COMPARISON")
    print("=" * 60)

    try:
        from sklearn.ensemble import (
            AdaBoostClassifier,
            GradientBoostingClassifier,
            RandomForestClassifier,
        )
        from sklearn.metrics import accuracy_score
    except ImportError:
        print("  sklearn not installed, skipping comparison.")
        print()
        return

    X, y = make_classification_data(n_samples=500)
    y_01 = (y + 1) // 2
    X_train, X_test, y_train, y_test = train_test_split(X, y)
    X_train_01, X_test_01, y_train_01, y_test_01 = train_test_split(X, y_01)

    ada_ours = AdaBoostScratch(n_estimators=50)
    ada_ours.fit(X_train, y_train)
    print(f"  Our AdaBoost:      {ada_ours.accuracy(X_test, y_test):.3f}")

    ada_sk = AdaBoostClassifier(n_estimators=50, random_state=42, algorithm="SAMME")
    ada_sk.fit(X_train_01, y_train_01)
    print(f"  sklearn AdaBoost:  {accuracy_score(y_test_01, ada_sk.predict(X_test_01)):.3f}")

    rf = RandomForestClassifier(n_estimators=100, random_state=42)
    rf.fit(X_train_01, y_train_01)
    print(f"  sklearn RF:        {accuracy_score(y_test_01, rf.predict(X_test_01)):.3f}")

    gb = GradientBoostingClassifier(n_estimators=100, random_state=42)
    gb.fit(X_train_01, y_train_01)
    print(f"  sklearn GBM:       {accuracy_score(y_test_01, gb.predict(X_test_01)):.3f}")

    print()


if __name__ == "__main__":
    demo_adaboost()
    demo_gradient_boosting()
    demo_learning_rate_effect()
    demo_bagging()
    demo_stacking()
    demo_comparison()
    demo_sklearn_comparison()
    print("All ensemble demos complete.")
You've finished the free half of ML Fundamentals

Most people stop here. Don't be most people.

The next 10 lessons are where Phase 02 turns from understanding into ability: the harder builds, the production patterns and the quizzes that prove you've actually got it. Unlock the full AI Expert Masterclass: all 542 expert lessons, every line of code, every phase.

  1. 12Hyperparameter Tuning
  2. 13ML Pipelines
  3. 14Naive Bayes
  4. 15Time Series Fundamentals
  5. + 6 more in this phase, and 542 across the whole masterclass