๐ Batch Normalization: Training & Inference
161 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# ๐ Batch Normalization: Training & Inference ## 1. ๐ฏ Learning Objectives - Explain why batch normalization accelerates training - Distinguish training behavior (batch stats) from inference (running averages) - Compare batch norm with layer norm and instance norm ## 2.
๐ Batch Normalization: Training & Inference
1. ๐ฏ Learning Objectives
- Explain why batch normalization accelerates training
- Distinguish training behavior (batch stats) from inference (running averages)
- Compare batch norm with layer norm and instance norm
2. ๐ Core Content
3.1 Internal Covariate Shift
The distribution of layer inputs changes during training because previous layers' parameters change. This forces layers to continuously adapt to shifting input distributions, slowing training.
3.2 Batch Normalization Algorithm
Training: Normalize using batch mean ฮผ_B and variance ฯยฒ_B, then scale/shift with learnable ฮณ, ฮฒ. Inference: Use running averages of ฮผ and ฯยฒ computed during training.
3.3 Benefits
- Allows higher learning rates (less sensitivity to initialization)
- Reduces need for dropout (provides regularization)
- Accelerates convergence (2-10ร faster)
- Reduces vanishing gradient problem
3.4 Normalization Variants
| Type | Normalizes Across | Use Case |
|---|---|---|
| Batch Norm | Batch dimension | CNNs, MLPs |
| Layer Norm | Feature dimension | RNNs, Transformers |
| Instance Norm | Single sample | Style transfer |
| Group Norm | Groups of channels | Small batch sizes |