Ensemble Methods: Bagging & Random Forest

🎯 Learning Objectives

• Explain why ensembles outperform individual models
• Describe bagging (Bootstrap Aggregating) and how it reduces variance
• Build and tune a Random Forest classifier
• Understand the difference between bagging and boosting
• Implement gradient boosting with sklearn

📋 Prerequisites

• Decision Trees — the base learner for most ensembles
• Bootstrap sampling — sampling with replacement
• Bias-variance tradeoff — ensembles reduce variance (bagging) or bias (boosting)

📖 Core Content

7.1 Intuition: The Wisdom of Crowds

7.2 Bagging: Bootstrap Aggregating

1. Bootstrap: Create B new training sets by sampling with replacement from the original dataset (each about 63% unique, 37% duplicated or missing).
2. Aggregate: Train a model on each bootstrap sample, then average (regression) or majority vote (classification).

python
Run Inline
# runnable
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = make_classification(n_samples=500, n_features=20, random_state=42)
# Single tree
tree = DecisionTreeClassifier(random_state=42)
tree_scores = cross_val_score(tree, X, y, cv=5)
print(f"Single tree: {tree_scores.mean():.3f} ± {tree_scores.std():.3f}")
# Bagged ensemble
bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(),  # Note: use 'estimator' not 'base_estimator' in newer sklearn
    n_estimators=100,
    max_samples=0.8,
    bootstrap=True,
    random_state=42
)
bag_scores = cross_val_score(bag, X, y, cv=5)
print(f"Bagging (100 trees): {bag_scores.mean():.3f} ± {bag_scores.std():.3f}")

7.3 Random Forest

• n_estimators: number of trees (higher is better, diminishing returns after ~100-500)
• max_features: number of features considered at each split (default: sqrt(n) for classification, n/3 for regression)
• max_depth: max tree depth (default: unlimited — trees grow fully)
• min_samples_leaf: minimum samples per leaf (default: 1 for classification)
• oob_score: use Out-Of-Bag samples as internal validation

python
Run Inline
# runnable
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
wine = load_wine()
X, y = wine.data, wine.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=10,
    min_samples_leaf=2,
    max_features='sqrt',
    oob_score=True,
    random_state=42
)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
print(f"Test accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"OOB score: {rf.oob_score_:.3f}")
# Feature importance
for name, imp in sorted(zip(wine.feature_names, rf.feature_importances_),
                         key=lambda x: x[1], reverse=True)[:5]:
    print(f"  {name}: {imp:.3f}")

7.4 Out-of-Bag (OOB) Evaluation

7.5 Boosting: Sequential Ensemble

python
Run Inline
# runnable
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import numpy as np
X, y = make_classification(n_samples=500, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
gb = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    min_samples_leaf=5,
    subsample=0.8,  # Stochastic: use 80% of data per tree
    random_state=42
)
gb.fit(X_train, y_train)
y_pred = gb.predict(X_test)
print(f"GB test accuracy: {accuracy_score(y_test, y_pred):.3f}")

7.6 Comparison: Bagging vs Boosting

7.7 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Too Few Trees

Pitfall 2: Overfitting with Boosting

Pitfall 3: Ignoring max_features in RF

📝 Practice Questions

Q1: What fraction of original data appears in a bootstrap sample?

About 63.2% (approximately 1 - 1/e). The probability a specific example is NOT selected in a sample of size m (with replacement) is (1 - 1/m)^m → 1/e ≈ 36.8%. So 63.2% appear at least once. Q2: Why does random forest outperform a single decision tree?

1. Variance reduction: Averaging B independent trees reduces variance by factor ≈ B (but trees aren't fully independent)
2. Feature randomization: Decorrelates trees further
3. Robustness: Less sensitive to noise and outliers

A single decision tree can change completely with minor data changes. RF averages over this instability. Q3: What is the difference between bagging and pasting?

• Bagging: Bootstrap sampling (with replacement) — some examples appear multiple times, some appear 0 times.
• Pasting: Sampling without replacement — each example appears at most once per sample. Pasted samples are smaller subsets.

Bagging is more common because it creates more diverse training sets. Q4: Your RF has training accuracy 0.99 and test accuracy 0.85. What do you do?

The gap (0.14) suggests overfitting. Solutions:

1. Reduce max_depth (limit tree growth)
2. Increase min_samples_leaf
3. Increase n_estimators (more averaging reduces variance)
4. Reduce max_features (more randomness)
5. Increase data if possible

The OOB score should be close to test score — if OOB is also 0.99, overfitting is confirmed. Q5: Explain how gradient boosting works in one sentence.

Gradient boosting trains trees sequentially where each new tree predicts the negative gradient (residual errors) of the loss function with respect to the current ensemble's predictions. Q6: When would you choose Gradient Boosting over Random Forest?

Choose Gradient Boosting when:

1. You need the highest possible accuracy (GB tends to win competitions)
2. Data is not too large (< 100k rows)
3. You have time for hyperparameter tuning
4. You're using shallow trees (weak learners)

Choose Random Forest when:

1. Simplicity and robustness matter
2. Data is very large (RF parallelizes easily)
3. You worry about overfitting (RF is more robust)
4. Quick baseline needed Q7: What is subsample in gradient boosting?

subsample (default 1.0) specifies the fraction of training data used per tree. Value 0.8 means each tree uses 80% of data (randomly selected). This adds randomness (like bagging) and helps prevent overfitting. Called Stochastic Gradient Boosting. Q8: Implement a Random Forest regressor for the Boston housing dataset.

python
Run Inline
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import numpy as np

housing = fetch_california_housing()
X, y = housing.data, housing.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

rf = RandomForestRegressor(n_estimators=200, max_depth=15, random_state=42)
rf.fit(X_train, y_train)

y_pred = rf.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f"RMSE: ${rmse:.3f}k")

Q9: What is a "weak learner" in boosting context?

A weak learner is a model that performs slightly better than random (e.g., accuracy > 50% for binary classification). In boosting, weak learners are typically decision stumps (depth 1 trees) or shallow trees (depth 2-3). The magic of boosting is combining many weak learners into one strong ensemble. Q10: How does early stopping work in gradient boosting?

Monitor validation error after each tree is added. If validation error hasn't improved for n_iter_no_change rounds, stop training. This prevents overfitting and saves computation. sklearn's GradientBoostingClassifier supports this with n_iter_no_change and validation_fraction parameters. Q11: Why does RF not require feature scaling?

Because decision trees (the base learners) are threshold-based, not distance-based. A tree tests "is feature > t?" regardless of the feature's scale. Doubling a feature's scale just doubles the threshold — the split point adapts. So RF works with raw data without any scaling. Q12: Compare bagging with pasting.

Bagging is generally preferred because the extra diversity helps ensembles more.

🔗 Cross-References

• Next Topic: Support Vector Machines
• Related: Decision Trees — base learner
• Related: Boosting Deep Dive — detailed gradient boosting
• External: IITM BSCS2004 Week 7, Hands-On ML Ch. 7, ISLR Ch. 8 Join Discord PreviousDecision TreesNextEnsemble Boosting