🛡️ Regularization

499 words

2 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# 🛡️ Regularization ## 1. 🎯 Learning Objectives - Explain L1 vs L2 regularization: effect on weights, sparsity - Implement dropout: forward pass with mask, inverted dropout - Explain batch normalization: training vs inference behavior - Apply early stopping and data augmentation - Diagnose overfitting and apply ap...

🛡️ Regularization

1. 🎯 Learning Objectives

Explain L1 vs L2 regularization: effect on weights, sparsity
Implement dropout: forward pass with mask, inverted dropout
Explain batch normalization: training vs inference behavior
Apply early stopping and data augmentation
Diagnose overfitting and apply appropriate regularization

2. 📖 Core Content

3.1 L2 Regularization (Weight Decay)

Adds squared magnitude of weights to loss:

L = L_0 + \frac{\lambda}{2}||w||_2^2 = L_0 + \frac{\lambda}{2}\sum w_i^2

Gradient:

\frac{\partial L}{\partial w_i} = \frac{\partial L_0}{\partial w_i} + \lambda w_i

Update:

w_i \leftarrow w_i - \eta(\frac{\partial L_0}{\partial w_i} + \lambda w_i) = w_i(1-\eta\lambda) - \eta\frac{\partial L_0}{\partial w_i}

The

w_i(1-\eta\lambda)

term is "weight decay" — it shrinks weights each step.

3.2 L1 Regularization (Lasso)

L = L_0 + \lambda|w|_1 = L_0 + \lambda\sum |w_i|

Effect: Produces sparse weights (many weights become exactly zero). Good for feature selection.

3.3 L1 vs L2 Comparison

Property	L2	L1
Penalty	$\sum w_i^2$	$\sum
Gradient	$\lambda w_i$	$\lambda \cdot \text{sign}(w_i)$
Weights	Small but non-zero	Sparse (many zero)
Best for	General regularization	Feature selection
Differentiable at 0	Yes (smooth)	No (sharp)

3.4 Dropout

During training, randomly "drop" (set to 0) a fraction p of neurons in a layer. This prevents co-adaptation — networks learn redundant representations. Forward pass (train):

y = f(Wx) \odot m \quad \text{where } m_i \sim \text{Bernoulli}(1-p)

Inverted Dropout (common implementation): During training:

y = \frac{1}{1-p} \cdot f(Wx) \odot m

During inference:

y = f(Wx)

(no scaling needed) Why it works: Creates an ensemble of exponentially many subnetworks.

3.5 Batch Normalization

Normalizes layer inputs to have zero mean and unit variance:

\mu_B = \frac{1}{m}\sum_{i=1}^m x_i

\sigma_B^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2

\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

y_i = \gamma\hat{x}_i + \beta

Where

\gamma

and

\beta

are learnable parameters. Training: Use batch statistics

\mu_B, \sigma_B^2

Inference: Use running averages of

\mu

and

\sigma^2

computed during training

3.6 Early Stopping

Monitor validation loss. Stop training when validation loss stops improving (with patience).


text
best_val_loss = infinity
patience_counter = 0
for epoch in range(max_epochs):
    train_loss = train_one_epoch()
    val_loss = evaluate()
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        save_model()
        patience_counter = 0
    else:
        patience_counter += 1
        if patience_counter >= patience:
            break

3.7 Data Augmentation

Generate more training data through label-preserving transformations:

Images: rotation, flip, crop, color jitter, noise
Text: synonym replacement, back-translation
Audio: time stretching, pitch shifting, noise addition

4. 📝 Practice Questions

Q1: With L2 regularization λ=0.01, learning rate η=0.1, and weight w=0.5, compute one update step (assume gradient from loss = 0.1).

Answer: Gradient = 0.1 + 0.01×0.5 = 0.105. w_new = 0.5 - 0.1×0.105 = 0.5 - 0.0105 = 0.4895. Weight decay factor: (1-0.1×0.01) = 0.999. Q2: With dropout p=0.5 and layer output [2, 3, 1, 4], what is the training output using inverted dropout?

Answer: Random mask: [1,0,1,0] (example). Output: (1/0.5)×[2×1, 3×0, 1×1, 4×0] = 2×[2,0,1,0] = [4,0,2,0]. Join Discord PreviousOptimization MethodsNextCNN Operations