Regularization Theory

🎯 Learning Objectives

• Derive Ridge and Lasso from constrained optimization
• Explain the geometry of L1 vs L2 regularization
• Understand the Bayesian interpretation (prior distributions)
• Implement cross-validated regularized models

📖 Core Content

1.1 Intuition: Why Regularization Works

1.2 Ridge Regression as Constrained Optimization

1.3 Lasso Geometry

1.4 Bayesian Interpretation

• Ridge = $\theta \sim \mathcal{N}(0, \sigma^2/\alpha)$ — Gaussian prior on weights
• Lasso = $\theta \sim \text{Laplace}(0, 1/\alpha)$ — Laplace prior on weights The Gaussian prior concentrates probability near zero but allows any value. The Laplace prior has a sharp peak at zero and heavy tails — this gives the sparsity property.

1.5 Bias-Variance Decomposition for Ridge

1.6 Implementation

python
Run Inline
# runnable
import numpy as np
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=200, n_features=20, noise=0.5, random_state=42)
# Ridge with built-in CV
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 20), cv=5)
ridge_cv.fit(X, y)
print(f"Ridge best α: {ridge_cv.alpha_:.3f}")
# Lasso with built-in CV
lasso_cv = LassoCV(alphas=np.logspace(-3, 3, 20), cv=5, random_state=42)
lasso_cv.fit(X, y)
print(f"Lasso best α: {lasso_cv.alpha_:.3f}")
print(f"Non-zero coefficients: {np.sum(lasso_cv.coef_ != 0)} / {len(lasso_cv.coef_)}")
# Elastic Net with built-in CV
elastic_cv = ElasticNetCV(l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9],
                          alphas=np.logspace(-3, 3, 20), cv=5, random_state=42)
elastic_cv.fit(X, y)
print(f"Elastic Net best α: {elastic_cv.alpha_:.3f}, l1_ratio: {elastic_cv.l1_ratio_:.2f}")

📝 Practice Questions

Q1: Show that Ridge has a closed-form solution.

$J(\theta) = (y - X\theta)^T(y - X\theta) + \alpha \theta^T \theta$

Set gradient to 0: $\nabla = -2X^T(y - X\theta) + 2\alpha\theta = 0$ $-X^T y + X^T X \theta + \alpha\theta = 0$ $(X^T X + \alpha I)\theta = X^T y$ $\theta = (X^T X + \alpha I)^{-1} X^T y$

This is always invertible because adding $\alpha I$ makes the matrix positive definite (even if $X^T X$ was singular). Q2: Why can't Lasso be solved in closed form?

The L1 penalty $\sum |\theta_j|$ is not differentiable at $\theta_j = 0$. There's a "kink" at zero where the derivative doesn't exist. This requires iterative optimization methods like coordinate descent or subgradient methods. Q3: Prove that Ridge reduces the variance of coefficient estimates.

From the closed form $\hat{\theta}_{ridge} = (X^T X + \alpha I)^{-1} X^T y$: $Var(\hat{\theta}_{ridge}) = \sigma^2 (X^T X + \alpha I)^{-1} X^T X (X^T X + \alpha I)^{-1}$

For orthogonal X (where $X^T X = I$): $Var(\hat{\theta}_{ridge}) = \frac{\sigma^2}{(1+\alpha)^2}$

Compare to OLS: $Var(\hat{\theta}_{OLS}) = \sigma^2$. The Ridge variance is smaller by factor $1/(1+\alpha)^2$. Join Discord PreviousCourse OverviewNextNaive Bayes