Regularization & Overfitting

🎯 Learning Objectives

• Explain how regularization prevents overfitting and when to use each type
• Implement Ridge (L2), Lasso (L1), and Elastic Net regression
• Use cross-validation to find optimal regularization strength
• Build a complete ML pipeline in sklearn including preprocessing and modeling
• Diagnose and fix overfitting systematically

📋 Prerequisites

• Linear & Logistic Regression — the models we regularize
• Model Evaluation — cross-validation, bias-variance tradeoff
• Feature Scaling — critical for regularized models

📖 Core Content

12.1 Intuition: Keeping the Model Humble

12.2 Ridge Regression (L2 Regularization)

• $\alpha = 0$: Standard linear regression (no regularization)
• $\alpha \to \infty$: All coefficients approach 0 (underfitting) Properties:
• Shrinks coefficients toward 0 (but never exactly 0)
• Handles multicollinearity well (distributes importance among correlated features)
• Has a closed-form solution: $\theta = (X^T X + \alpha I)^{-1} X^T y$

python
Run Inline
# runnable
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Ridge with different alphas
for alpha in [0, 0.1, 1, 10, 100]:
    ridge = Ridge(alpha=alpha)
    ridge.fit(X_train, y_train)
    train_score = ridge.score(X_train, y_train)
    test_score = ridge.score(X_test, y_test)
    print(f"α={alpha:5.1f}: Train R²={train_score:.3f}, Test R²={test_score:.3f}")

12.3 Lasso Regression (L1 Regularization)

• Forces some coefficients to exactly 0 (feature selection!)
• Produces sparse models (fewer features used)
• Good when you believe only a few features matter

python
Run Inline
# runnable
from sklearn.linear_model import Lasso
from sklearn.datasets import load_diabetes
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)
print(f"Features used: {np.sum(lasso.coef_ != 0)} / {len(lasso.coef_)}")
print(f"Coefficients: {lasso.coef_}")
print(f"Test R²: {lasso.score(X_test, y_test):.3f}")

12.4 Worked Example 1: Ridge vs Lasso by Hand

12.5 Elastic Net: Best of Both Worlds

• $r = 1$: Pure Lasso
• $r = 0$: Pure Ridge
• $r = 0.5$: Equal mixture Elastic Net is preferred when:
• You have more features than samples (p > n)
• Features are grouped (Lasso picks one, Ridge picks all, Elastic Net picks groups)
• You want Ridge's stability + Lasso's sparsity

python
Run Inline
# runnable
from sklearn.linear_model import ElasticNet
from sklearn.model_selection import GridSearchCV
elastic = ElasticNet()
param_grid = {
    'alpha': [0.001, 0.01, 0.1, 1, 10],
    'l1_ratio': [0.1, 0.3, 0.5, 0.7, 0.9]
}
grid = GridSearchCV(elastic, param_grid, cv=5, scoring='r2')
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
print(f"Best CV R²: {grid.best_score_:.3f}")

12.6 Comparing Regularization Methods

12.7 Early Stopping (Gradient Descent Regularization)

python
Run Inline
# runnable
from sklearn.linear_model import SGDRegressor
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Without early stopping
sgd = SGDRegressor(max_iter=1000, tol=1e-3, random_state=42, penalty=None)
sgd.fit(X_train, y_train)
print(f"No regularization: Test R² = {sgd.score(X_test, y_test):.3f}")
# With early stopping (fewer iterations = simpler model)
sgd_early = SGDRegressor(max_iter=10, tol=None, random_state=42, penalty=None)
sgd_early.fit(X_train, y_train)
print(f"Early stopping (iter=10): Test R² = {sgd_early.score(X_test, y_test):.3f}")

12.8 The ML Pipeline

python
Run Inline
# runnable
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
import numpy as np
# Build pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('poly', PolynomialFeatures(degree=2, include_bias=False)),
    ('ridge', Ridge(alpha=1.0))
])
# Use as a single estimator
scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring='r2')
print(f"Pipeline CV R²: {scores.mean():.3f} ± {scores.std():.3f}")
# Grid search over pipeline params
from sklearn.model_selection import GridSearchCV
param_grid = {
    'poly__degree': [1, 2, 3],
    'ridge__alpha': [0.01, 0.1, 1, 10]
}
grid = GridSearchCV(pipeline, param_grid, cv=5)
grid.fit(X_train, y_train)
print(f"Best pipeline: {grid.best_params_}")
print(f"Test R²: {grid.score(X_test, y_test):.3f}")

12.9 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Not Scaling Before Regularization

Pitfall 2: Choosing α Too Large

Pitfall 3: Lasso with Highly Correlated Features

📝 Practice Questions

Q1: Ridge shrinks coefficients toward 0. Why would we want to do this?

Large coefficients mean the model relies heavily on individual features. If those features change slightly (noise), predictions change dramatically. Shrinking coefficients makes the model more robust — each feature contributes modestly, reducing variance. This is the classic bias-variance tradeoff: we accept a small bias increase for a large variance decrease. Q2: With α=0, Ridge is equivalent to what?

Standard Linear Regression (OLS). No regularization is applied. The coefficients minimize only the MSE. Q3: Lasso gives coefficients [0, 2.5, 0, 0.3, 0, 0]. What does this tell you?

Only 2 features out of 6 are used (features 1 and 3 with values 2.5 and 0.3). Lasso performed feature selection — the other 4 features had coefficients shrunk to exactly 0. This model is sparse and likely simpler to deploy. Q4: Why does Lasso produce zero coefficients but Ridge doesn't?

The L1 penalty's derivative is constant (±α), so the optimization pushes coefficients all the way to 0 when they're small. The L2 penalty's derivative is 2αθ, which approaches 0 as θ approaches 0 — so Ridge keeps shrinking but never fully zeros out.

Geometrically: Lasso's constraint region is a diamond (with corners at axis intersections), Ridge's is a circle (no corners). Q5: You have 10 features, 50 samples, and high multicollinearity. Which regularization?

Ridge — it handles multicollinearity well (distributes coefficients among correlated features) and works with p < n (10 < 50). Elastic Net would also work. Lasso would be unstable with correlated features. Q6: Implement a pipeline with StandardScaler, PolynomialFeatures, and Ridge.

python
Run Inline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import Ridge

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('poly', PolynomialFeatures(degree=2)),
    ('model', Ridge(alpha=1.0))
])
pipeline.fit(X_train, y_train)

The pipeline ensures that:

1. Scaling is fit on training data and applied consistently
2. Polynomial features are created after scaling
3. Ridge regression is the final estimator Q7: What is the information-theoretic justification for regularization?

Occam's razor: simpler models are preferred. Regularization penalizes model complexity (large coefficients = complex model). This connects to:

• VC-dimension: Regularized models have lower effective VC-dimension
• Bayesian prior: L2 regularization = Gaussian prior on weights, L1 = Laplace prior
• Minimum Description Length: Regularization prefers models that compress the data better Q8: With 100 features but only 2 are truly relevant, which regularizer?

Lasso (L1) — it will drive the 98 irrelevant feature coefficients to 0. Ridge would keep them small but non-zero, making the model harder to interpret. Lasso naturally performs feature selection for sparse solutions. Q9: In Elastic Net, what does l1_ratio=0.3 mean?

30% L1 penalty + 70% L2 penalty. The model is closer to Ridge than Lasso. It will shrink coefficients (L2) but also push some to zero (L1), handling correlated features better than pure Lasso. Q10: How do you choose α in Ridge regression?

Use cross-validation with RidgeCV or GridSearchCV:

python
Run Inline
from sklearn.linear_model import RidgeCV
ridge_cv = RidgeCV(alphas=[0.001, 0.01, 0.1, 1, 10, 100], cv=5)
ridge_cv.fit(X_train, y_train)
print(f"Best α: {ridge_cv.alpha_}")

The optimal α balances bias and variance for your specific dataset. Q11: Why is scaling important for L1 regularization?

The L1 penalty sums |θⱼ|. If feature A has values [0.1, 0.2] and feature B has [100, 200], feature B's coefficient will be ~1000× smaller to have the same effect. With L1, a small coefficient for B gets zeroed out easily, while A's larger coefficient survives. This biases the model against features with small scales. Q12: What happens if you apply Lasso to one-hot encoded categorical features?

One-hot encoding creates perfectly correlated dummy variables (sum to 1). Lasso will arbitrarily pick one category and zero out the rest, losing information. Solution: use Elastic Net (which groups correlated features) or drop one category before Lasso.

🔗 Cross-References

• Previous Topic: Model Evaluation & Cross-Validation
• Related: Polynomial Regression — where we first saw overfitting
• Related: Feature Selection — Lasso is a feature selection method
• External: IITM BSCS2004 Week 12, Hands-On ML Ch. 4, ISLR Ch. 6 Join Discord PreviousModel Evaluation & CVNextFeature Engineering