Hyperparameter Tuning

🎯 Learning Objectives

• Compare grid search, random search, and Bayesian optimization
• Implement hyperparameter tuning with cross-validation
• Use Optuna for advanced optimization
• Apply early stopping and pruning

📖 Core Content

5.1 Search Strategies

5.2 Grid vs Random Search

python
Run Inline
# runnable
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from scipy.stats import randint, uniform
iris = load_iris()
X, y = iris.data, iris.target
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, None],
    'min_samples_leaf': [1, 2, 5]
}
# Grid Search (3 × 3 × 3 = 27 combinations)
grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5)
grid.fit(X, y)
print(f"Grid best: {grid.best_params_}")
# Random Search (sample 10 combinations)
param_dist = {
    'n_estimators': randint(50, 300),
    'max_depth': randint(3, 20),
    'min_samples_leaf': randint(1, 10)
}
random = RandomizedSearchCV(RandomForestClassifier(), param_dist, n_iter=10, cv=5, random_state=42)
random.fit(X, y)
print(f"Random best: {random.best_params_}")

5.3 Bayesian Optimization with Optuna

python
Run Inline
# runnable
# Note: Requires optuna
# import optuna
# from sklearn.ensemble import RandomForestClassifier
# from sklearn.model_selection import cross_val_score
#
# def objective(trial):
#     params = {
#         'n_estimators': trial.suggest_int('n_estimators', 50, 300),
#         'max_depth': trial.suggest_int('max_depth', 3, 20),
#         'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
#         'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2'])
#     }
#     model = RandomForestClassifier(**params, random_state=42)
#     score = cross_val_score(model, X, y, cv=5).mean()
#     return score
#
# study = optuna.create_study(direction='maximize')
# study.optimize(objective, n_trials=50)
# print(f"Optuna best: {study.best_params}")

📝 Practice Questions

Q1: Why is random search better than grid search?

Bergstra & Bengio (2012) showed empirically: not all hyperparameters matter equally. Grid search wastes trials on unimportant parameters. Random search samples each dimension independently, so it discovers optimal values for important parameters with fewer total trials. For the same compute budget, random search finds better models. Q2: How does Bayesian optimization work?

It builds a probabilistic model (Gaussian Process) of the objective function f(hyperparams → validation score). After each trial, it updates the model and uses an acquisition function (expected improvement) to select the next most promising parameters. It's "intelligent search" — learns from previous trials to focus on promising regions. Q3: What is trial pruning in Optuna?

During training, if a trial's intermediate results are clearly worse than other trials, Optuna can stop (prune) it early. Example: if after 10 epochs of 100, validation accuracy is 50% while others achieved 70%, stop this trial — it won't catch up. Pruning saves compute for more promising configurations. Join Discord PreviousModel DeploymentNextModel Interpretation