Model Evaluation & Cross-Validation

🎯 Learning Objectives

• Explain the bias-variance tradeoff and diagnose underfitting vs overfitting
• Implement k-fold cross-validation for reliable model evaluation
• Use learning curves to determine if more data would help
• Perform hyperparameter tuning with grid search and randomized search
• Select between competing models using statistical tests

📋 Prerequisites

• All previous MLF topics — this is the evaluation framework
• Basic statistics — variance, standard deviation, hypothesis testing
• Overfitting concepts — from polynomial regression and decision trees

📖 Core Content

11.1 Intuition: How Do We Know If Our Model Is Good?

11.2 The Three-Way Split

11.3 Cross-Validation

1. Split data into k equal folds
2. For each fold i:
   • Train on all folds except fold i
   • Evaluate on fold i
3. Report mean and std of k evaluation scores

python
Run Inline
# runnable
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
import numpy as np
iris = load_iris()
X, y = iris.data, iris.target
model = LogisticRegression(max_iter=200)
# 5-fold CV
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"5-fold CV scores: {scores}")
print(f"Mean accuracy: {scores.mean():.3f} ± {scores.std():.3f}")
# Custom CV with shuffling
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
scores_shuffled = cross_val_score(model, X, y, cv=kfold, scoring='accuracy')
print(f"Shuffled 5-fold: {scores_shuffled.mean():.3f} ± {scores_shuffled.std():.3f}")

11.4 Worked Example 1: 5-Fold CV for Model Selection

11.5 Bias-Variance Tradeoff

• Add more features
• Increase model complexity (higher polynomial degree)
• Reduce regularization
• Try a more powerful algorithm Fixing Variance (overfitting):
• Add more training data
• Reduce model complexity
• Increase regularization
• Add feature selection

11.6 Learning Curves

python
Run Inline
# runnable
from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import matplotlib.pyplot as plt
train_sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=5),
    X, y, cv=5, scoring='accuracy',
    train_sizes=np.linspace(0.1, 1.0, 10)
)
train_mean = np.mean(train_scores, axis=1)
val_mean = np.mean(val_scores, axis=1)
plt.plot(train_sizes, train_mean, 'o-', label='Training')
plt.plot(train_sizes, val_mean, 'o-', label='Validation')
plt.xlabel('Training examples')
plt.ylabel('Accuracy')
plt.title('Learning Curve')
plt.legend()
plt.grid(True)
plt.show()

• High bias: Both curves converge to a low score. Adding data won't help — need a better model.
• High variance: Large gap between training and validation curves. Adding data will help.
• Good fit: Both curves converge to a high score with a small gap.

11.7 Hyperparameter Tuning

python
Run Inline
# runnable
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_wine
wine = load_wine()
X, y = wine.data, wine.target
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, None],
    'min_samples_leaf': [1, 2, 5]
}
grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    verbose=1
)
grid.fit(X, y)
print(f"Best params: {grid.best_params_}")
print(f"Best CV score: {grid.best_score_:.3f}")
print(f"Test score: {grid.score(X, y):.3f}")

python
Run Inline
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
param_dist = {
    'n_estimators': randint(50, 500),
    'max_depth': randint(3, 20),
    'min_samples_leaf': randint(1, 10),
    'max_features': uniform(0.5, 0.5)
}
random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_dist,
    n_iter=50,  # Try 50 random combinations
    cv=5,
    random_state=42
)

11.8 Model Selection Strategies

11.9 Comparing Models: The Statistical Perspective

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Data Leakage

Pitfall 2: Optimism Bias from Tuning on Test

Pitfall 3: Ignoring Variance of CV Estimates

📝 Practice Questions

Q1: You observe training error = 0.01 and validation error = 0.30. What's the problem?

Overfitting (high variance). The model performs nearly perfectly on training data but poorly on validation data. Solutions: increase regularization, reduce model complexity, add more training data, or simplify by reducing features. Q2: Both training and validation errors are high (0.40 and 0.45). What's the problem?

Underfitting (high bias). The model is too simple to capture the underlying pattern. Solutions: use a more complex model, add more features, reduce regularization, or try a different algorithm entirely. Q3: Why use stratified k-fold for classification?

Standard k-fold might create folds with very different class proportions (e.g., all positive examples in one fold). Stratified k-fold ensures each fold has approximately the same class distribution as the full dataset. This gives more reliable and less variable CV scores, especially for imbalanced data. Q4: With 1,000 examples, which CV strategy is more efficient: 5-fold or 10-fold?

5-fold CV: Train on 800, test on 200. 10-fold: Train on 900, test on 100. 5-fold is more computationally efficient (5 models vs 10) but has slightly higher bias (less training data per model). 10-fold gives a less biased estimate but costs more. For 1000 examples, 5-fold is a good default. Q5: Your learning curves show training and validation error converging to 0.45. What does this mean?

The model is underfitting (high bias). Both curves converge to the same high error. Adding more data won't help — the model's capacity is insufficient to capture the pattern. The fix is to increase model complexity. Q6: How many models does 5-fold CV with 10 hyperparameter combinations train?

5 folds × 10 combinations = 50 models. Each combination is evaluated on all 5 folds. This is why grid search can be expensive — be strategic about what you search. Q7: What is the "one standard error" rule?

When comparing models via CV, choose the simplest model whose score is within one standard error of the best model. This trades off a small performance drop for significantly simpler/more interpretable model. It's a practical application of Occam's razor. Q8: Explain why LOOCV has high variance.

LOOCV trains m models (one per data point). Each training set differs by just 1 example, so the m models are highly correlated. The average of highly correlated estimates has the same variance as the individual estimates — no variance reduction from averaging. With k-fold (k=5 or 10), the training sets overlap less, giving more independent models and thus lower variance in the estimate. Q9: When would you use RandomizedSearchCV over GridSearchCV?

RandomizedSearchCV is better when:

1. The hyperparameter space is large (>10 parameters)
2. You have limited compute budget
3. Some parameters are continuous (distributions, not discrete values)

Empirically, random search finds good hyperparameters faster than grid search because not all parameters matter equally. Q10: After tuning on validation data, should you then train on both train+val?

Yes! This is standard practice:

1. Find optimal hyperparameters via CV on training set
2. Retrain model on full training set (train + validation combined)
3. Evaluate ONCE on the held-out test set

The final model uses all available data for training. Q11: How does increasing k in k-fold CV affect bias and variance of the estimate?

• Higher k (e.g., 10-fold vs 5-fold): Lower bias (more training data per fold), but higher variance (more overlap between folds → correlated estimates)
• Lower k (e.g., 3-fold): Higher bias (less training data), but lower variance (less overlap)

k=5 or k=10 is the standard compromise. Q12: Your model's accuracy is 0.90 but the business needs 0.95. What's your plan?

1. Check if the gap is achievable: Is the Bayes error (irreducible) lower than 0.05?
2. Diagnose bias-variance: If underfitting, get better features or model. If overfitting, get more data.
3. Try more complex models: Ensemble methods, gradient boosting, neural networks.
4. Improve data quality: Clean errors, add features, collect more data.
5. Adjust threshold: Maybe the right threshold yields better business-relevant metrics.
6. Consider different metric: Maybe accuracy isn't the right metric — precision/recall might be more relevant.

🔗 Cross-References

• Next Topic: Regularization & Overfitting
• Related: Bias-Variance Tradeoff
• Related: [All MLF topics] — evaluation applies universally
• External: IITM BSCS2004 Week 11, Hands-On ML Ch. 2 (end-to-end project includes CV) Join Discord PreviousDimensionality ReductionNextRegularization