Boosting: AdaBoost, Gradient Boosting & XGBoost

🎯 Learning Objectives

• Explain how boosting converts weak learners to strong ensembles
• Implement AdaBoost and Gradient Boosting with sklearn
• Tune hyperparameters for gradient boosting
• Understand when boosting outperforms bagging

📖 Core Content

14.1 Intuition: Learning from Mistakes

14.2 AdaBoost

1. Initialize all sample weights $w_i = 1/m$
2. For $t = 1, 2, \dots, T$:
   • Train a weak learner $h_t(x)$ using current weights
   • Compute weighted error $\epsilon_t = \sum_{i: h_t(x_i) \neq y_i} w_i / \sum w_i$
   • Compute learner weight $\alpha_t = 0.5 \ln((1-\epsilon_t)/\epsilon_t)$
   • Update sample weights: $w_i \leftarrow w_i \cdot \exp(-\alpha_t y_i h_t(x_i))$
   • Normalize weights to sum to 1
3. Final prediction: $\hat{y} = \text{sign}(\sum \alpha_t h_t(x))$

python
Run Inline
# runnable
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=500, random_state=42)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),  # Stump = weak learner
    n_estimators=100,
    learning_rate=1.0,
    random_state=42
)
ada.fit(X, y)
print(f"AdaBoost accuracy: {ada.score(X, y):.3f}")

14.3 Gradient Boosting (GBM)

1. Initialize $F_0(x) = \bar{y}$ (constant prediction)
2. For $t = 1, 2, \dots, T$:
   • Compute residuals: $r_i = y_i - F_{t-1}(x_i)$ (for MSE loss)
   • Fit a tree $h_t(x)$ to predict $r_i$
   • Update: $F_t(x) = F_{t-1}(x) + \nu \cdot h_t(x)$ Where $\nu$ (learning_rate) controls how much each tree contributes.

python
Run Inline
# runnable
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=1000, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
gb = GradientBoostingClassifier(
    n_estimators=200,
    learning_rate=0.1,
    max_depth=3,
    subsample=0.8,
    min_samples_leaf=10,
    random_state=42
)
gb.fit(X_train, y_train)
print(f"GB test accuracy: {gb.score(X_test, y_test):.3f}")

14.4 XGBoost: Extreme Gradient Boosting

• Regularization (L1 and L2 on tree weights)
• Parallel processing (tree construction is parallelized)
• Handling missing values (learns default direction)
• Weighted quantile sketch (efficient approximate split finding)
• Cross-validation during training

python
Run Inline
# Note: xgboost may not be installed - this is reference code
# import xgboost as xgb
# model = xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, max_depth=3,
#                           reg_alpha=0.1, reg_lambda=1.0, subsample=0.8)
# model.fit(X_train, y_train)

14.5 Hyperparameter Tuning

📝 Practice Questions

Q1: How does AdaBoost assign weights to training examples?

Initially all weights are equal (1/m). After each round, misclassified examples have their weights increased (multiplied by e^α), and correctly classified ones have weights decreased. This forces the next learner to focus on hard examples. Q2: What's the difference between AdaBoost and Gradient Boosting?

AdaBoost reweights training examples; Gradient Boosting fits new models to the residuals of previous models. AdaBoost can use any weak learner; Gradient Boosting typically uses shallow trees. Gradient Boosting generalizes to any differentiable loss function. Q3: Why use shallow trees (max_depth=2-4) in gradient boosting?

Boosting relies on weak learners — models slightly better than random. Deep trees would overfit the residuals of each step, making the ensemble unstable. Shallow trees ensure each tree captures just a bit of the remaining pattern, building up complexity gradually. Q4: What does the learning rate do in gradient boosting?

It shrinks each tree's contribution. Learning rate = 1.0 means each tree fully corrects residuals. Learning rate = 0.1 means each tree corrects only 10% — so you need more trees but get a smoother, more robust model. Lower learning rate + more trees = better generalization. Join Discord PreviousBagging & Random ForestNextSupport Vector Machines