Neural Sync Active

Regularization Theory

Registry Synced

Regularization Theory

660 words

3 min read

Reading compass

Now · 🎯 Learning Objectives

Regularization Theory

🎯 Learning Objectives

Derive Ridge and Lasso from constrained optimization
Explain the geometry of L1 vs L2 regularization
Understand the Bayesian interpretation (prior distributions)
Implement cross-validated regularized models

📖 Core Content

1.1 Intuition: Why Regularization Works

Think of model coefficients like a budget. Without regularization, the model can spend its "coefficient budget" however it likes — putting all its money on one feature if that minimizes training error. Regularization imposes a tax on large coefficients. The model must now pay a penalty for spending too much on any one feature, forcing it to distribute its budget more evenly or eliminate features entirely.

1.2 Ridge Regression as Constrained Optimization

Standard linear regression: minimize MSE. Ridge regression: minimize MSE subject to

\sum \theta_j^2 \leq t

By Lagrangian duality, this is equivalent to:

\min_{\theta} MSE(\theta) + \alpha \sum \theta_j^2

The geometry: The constraint

\sum \theta_j^2 \leq t

is a circle/sphere in parameter space. The unconstrained minimum of MSE likely lies outside this circle. The constrained optimum is where the MSE contours first touch the circle — which is always on the boundary (not at the origin).

1.3 Lasso Geometry

Lasso constraint:

\sum |\theta_j| \leq t

— a diamond/square. The key difference: The diamond has corners at the axes. If the MSE contours touch at a corner, that coefficient is exactly zero. This is why Lasso produces sparse solutions. (Diagram)

1.4 Bayesian Interpretation

Ridge = $\theta \sim \mathcal{N}(0, \sigma^2/\alpha)$ — Gaussian prior on weights
Lasso = $\theta \sim \text{Laplace}(0, 1/\alpha)$ — Laplace prior on weights The Gaussian prior concentrates probability near zero but allows any value. The Laplace prior has a sharp peak at zero and heavy tails — this gives the sparsity property.

1.5 Bias-Variance Decomposition for Ridge

For Ridge regression with orthogonal features

X^T X = I

\hat{\theta}_{ridge} = \frac{\theta}{1 + \alpha}

Bias(\hat{\theta}_{ridge}) = -\frac{\alpha\theta}{1 + \alpha}

Var(\hat{\theta}_{ridge}) = \frac{\sigma^2}{(1 + \alpha)^2}

\alpha

increases: bias increases, variance decreases. The total MSE (bias² + variance) has a minimum at some intermediate

\alpha

1.6 Implementation


python
# runnable
import numpy as np
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=200, n_features=20, noise=0.5, random_state=42)
# Ridge with built-in CV
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 20), cv=5)
ridge_cv.fit(X, y)
print(f"Ridge best α: {ridge_cv.alpha_:.3f}")
# Lasso with built-in CV
lasso_cv = LassoCV(alphas=np.logspace(-3, 3, 20), cv=5, random_state=42)
lasso_cv.fit(X, y)
print(f"Lasso best α: {lasso_cv.alpha_:.3f}")
print(f"Non-zero coefficients: {np.sum(lasso_cv.coef_ != 0)} / {len(lasso_cv.coef_)}")
# Elastic Net with built-in CV
elastic_cv = ElasticNetCV(l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9],
                          alphas=np.logspace(-3, 3, 20), cv=5, random_state=42)
elastic_cv.fit(X, y)
print(f"Elastic Net best α: {elastic_cv.alpha_:.3f}, l1_ratio: {elastic_cv.l1_ratio_:.2f}")

📝 Practice Questions

Q1: Show that Ridge has a closed-form solution.

$J(\theta) = (y - X\theta)^T(y - X\theta) + \alpha \theta^T \theta$

Set gradient to 0: $\nabla = -2X^T(y - X\theta) + 2\alpha\theta = 0$ $-X^T y + X^T X \theta + \alpha\theta = 0$ $(X^T X + \alpha I)\theta = X^T y$ $\theta = (X^T X + \alpha I)^{-1} X^T y$

This is always invertible because adding $\alpha I$ makes the matrix positive definite (even if $X^T X$ was singular). Q2: Why can't Lasso be solved in closed form?

The L1 penalty $\sum |\theta_j|$ is not differentiable at $\theta_j = 0$ . There's a "kink" at zero where the derivative doesn't exist. This requires iterative optimization methods like coordinate descent or subgradient methods. Q3: Prove that Ridge reduces the variance of coefficient estimates.

From the closed form $\hat{\theta}_{ridge} = (X^T X + \alpha I)^{-1} X^T y$ : $Var(\hat{\theta}_{ridge}) = \sigma^2 (X^T X + \alpha I)^{-1} X^T X (X^T X + \alpha I)^{-1}$

For orthogonal X (where $X^T X = I$ ): $Var(\hat{\theta}_{ridge}) = \frac{\sigma^2}{(1+\alpha)^2}$

Compare to OLS: $Var(\hat{\theta}_{OLS}) = \sigma^2$ . The Ridge variance is smaller by factor $1/(1+\alpha)^2$ . Join Discord PreviousCourse OverviewNextNaive Bayes