🧱 ResNet & Skip Connections
333 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# 🧱 ResNet & Skip Connections ## 1. 🎯 Learning Objectives - Explain the degradation problem in very deep networks - Implement a residual block - Describe ResNet architectures (ResNet-18, 34, 50, 101, 152) - Understand bottleneck design for efficiency ## 2.
🧱 ResNet & Skip Connections
1. 🎯 Learning Objectives
- Explain the degradation problem in very deep networks
- Implement a residual block
- Describe ResNet architectures (ResNet-18, 34, 50, 101, 152)
- Understand bottleneck design for efficiency
2. 📖 Core Content
3.1 The Degradation Problem
Intuitively, deeper networks should perform at least as well as shallower ones — if we add identity layers, the deeper network should match the shallower one. But in practice, deeper networks had HIGHER training error. This is the degradation problem, NOT caused by overfitting.
3.2 Residual Block
Idea: Instead of learning H(x) directly, learn the residual F(x) = H(x) - x, then:
If identity is optimal (H(x) = x), the network can set F(x) = 0, which is easier than learning identity directly.
Forward pass: y = ReLU(F(x) + x) where F(x) = W₂·ReLU(W₁·x + b₁) + b₂
3.3 Why It Works
- Gradient highway: Gradient flows directly through the skip connection (the + x path), bypassing weight layers
- Easier optimization: Residual functions are closer to zero than direct mappings
- Ensemble effect: ResNets behave like ensembles of shallower networks
3.4 ResNet Architectures
| Architecture | Layers | Key Design | Parameters |
|---|---|---|---|
| ResNet-18 | 18 | Basic blocks (2×3×3 conv) | 11M |
| ResNet-34 | 34 | Basic blocks | 22M |
| ResNet-50 | 50 | Bottleneck blocks (1×1, 3×3, 1×1) | 26M |
| ResNet-101 | 101 | Bottleneck | 45M |
| ResNet-152 | 152 | Bottleneck | 60M |
3.5 Bottleneck Block
For deeper ResNets, use 3 layers: 1×1 (reduce channels) → 3×3 (spatial) → 1×1 (restore channels). This is more efficient than two 3×3 layers.
4. 📝 Practice Questions
Q1: A ResNet has a residual block with input 256 channels. The bottleneck reduces to 64 channels for 3×3 conv, then back to 256. How many parameters does this block save compared to two 3×3 convs with 256 channels?Answer: Bottleneck: 1×1(256→64) + 3×3(64→64) + 1×1(64→256) = 256×64 + 9×64×64 + 64×256 = 16,384 + 36,864 + 16,384 = 69,632 params. Two 3×3(256→256): 2 × 9 × 256 × 256 = 1,179,648 params. Bottleneck is ~17x more efficient! Join Discord PreviousCNN ArchitecturesNextRNN Variants