Learning Objectives

• Perform hyperparameter optimization
• Use GridSearchCV and RandomizedSearchCV
• Avoid overfitting during tuning

python
Run Inline
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
import numpy as np
# Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4],
    'max_features': ['sqrt', 'log2']
}
# Grid search
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='f1',
    n_jobs=-1,
    verbose=1
)
grid_search.fit(X_train, y_train)
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")
# Evaluate on test
best_model = grid_search.best_estimator_
test_score = best_model.score(X_test, y_test)
print(f"Test score: {test_score:.3f}")
print(f"Gap (possible overfitting): {grid_search.best_score_ - test_score:.3f}")

Q1: Grid search vs random search?

Grid: exhaustive search over all combinations. Random: sample from distribution. Random search is more efficient for high-dimensional spaces (>3 params). Q2: How to avoid overfitting during tuning?

Use nested cross-validation (inner CV for tuning, outer CV for evaluation). Keep separate hold-out test set. Don't tune on test data. Q3: What is Bayesian optimization?

Builds probabilistic model (Gaussian Process) of objective function. Balances exploration and exploitation. More efficient than grid/random for expensive evaluations. Q4: What is early stopping?

Stop training when validation performance stops improving. Prevents overfitting. Implemented in XGBoost, LightGBM, neural networks (patience parameter). Q5: How many hyperparameter combinations?

Start with coarse grid (few values each), then refine around best regions. Total combinations = product of choices. Keep under 100-200 for grid search. Join Discord PreviousMilestone 4: Model Training & EvaluationNextMilestone 6: Model Interpretation & Feature Importance