Ensemble Methods: Bagging, Boosting, Stacking, Voting

🎯 Learning Objectives

• Understand bagging (Random Forest) and boosting (XGBoost, LightGBM)
• Stack multiple models for improved performance
• Use voting classifiers for robust predictions
• Choose the right ensemble for different data types

📖 Core Content

1.1 Bagging: Bootstrap + Aggregation

python
Run Inline
# runnable
from sklearn.ensemble import RandomForestClassifier, BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
# Random Forest (bagging + random feature selection)
rf = RandomForestClassifier(
    n_estimators=100,     # Number of trees
    max_depth=10,         # Control overfitting
    max_features='sqrt',  # sqrt(n_features) per split
    min_samples_leaf=5    # Minimum samples per leaf
)

1.2 Boosting: Sequential Correction

python
Run Inline
# runnable
# XGBoost (most popular)
# import xgboost as xgb
# model = xgb.XGBClassifier(
#     n_estimators=100,
#     learning_rate=0.1,
#     max_depth=6,
#     subsample=0.8,
#     colsample_bytree=0.8,
#     reg_lambda=1.0  # L2 regularization
# )
# LightGBM (faster, leaf-wise growth)
# import lightgbm as lgb
# model = lgb.LGBMClassifier(
#     n_estimators=100,
#     learning_rate=0.1,
#     num_leaves=31,
#     subsample=0.8,
#     colsample_bytree=0.8
# )

1.3 Stacking

python
Run Inline
# runnable
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
base_models = [
    ('rf', RandomForestClassifier(n_estimators=50)),
    ('svm', SVC(probability=True)),
    ('lr', LogisticRegression())
]
meta_model = LogisticRegression()
stacking = StackingClassifier(
    estimators=base_models,
    final_estimator=meta_model,
    cv=5  # Base models trained with CV to prevent overfitting
)

1.4 Why This Matters

2. 📝 Practice Questions

Q1: Your Random Forest takes 5 minutes to train on 100K rows. For a time-sensitive application, name 3 ways to speed it up without dropping performance more than 2%.

1. Reduce n_estimators: From 500 to 100. Diminishing returns beyond ~100 trees.
2. Increase min_samples_leaf: From 1 to 10. Reduces tree depth.
3. Use max_features='log2': Fewer features considered per split.
4. Use subsample=0.5: Train each tree on 50% of data (bootstrapping with less data).
5. Set max_depth=15: Limit tree growth.
6. Switch to LightGBM: 5-10× faster than RF for large datasets. Join Discord PreviousCross-ValidationNextHandling Imbalanced Data