ResNet, Skip Connections, and Deep Network Design

🎯 Learning Objectives

• Understand the vanishing gradient problem in deep networks
• Explain how skip connections enable training of very deep networks
• Implement a residual block with bottleneck design
• Compare ResNet variants (18, 34, 50, 101, 152)

📋 Prerequisites

• CNN fundamentals
• Backpropagation and gradient flow

---

1. 📖 Core Content

1.1 The Degradation Problem

1.2 Residual Learning

$$H(x) = F(x) + x$$

python
Run Inline
# runnable
import torch.nn as nn
class BasicBlock(nn.Module):
    """Basic residual block for ResNet-18/34"""
    expansion = 1
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        # Skip connection (1×1 conv if dimensions differ)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )
    def forward(self, x):
        residual = self.shortcut(x)  # Skip connection
        out = nn.ReLU()(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += residual  # Element-wise addition
        out = nn.ReLU()(out)
        return out
class Bottleneck(nn.Module):
    """Bottleneck block for ResNet-50/101/152 (1×1 → 3×3 → 1×1)"""
    expansion = 4
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.conv3 = nn.Conv2d(out_channels, out_channels * 4, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_channels * 4)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels * 4:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels * 4, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels * 4)
            )
    def forward(self, x):
        residual = self.shortcut(x)
        out = nn.ReLU()(self.bn1(self.conv1(x)))
        out = nn.ReLU()(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += residual
        out = nn.ReLU()(out)
        return out

1.3 ResNet Architectures

1.4 Why Skip Connections Work

$$\frac{\partial L}{\partial x_l} = \frac{\partial L}{\partial x_L} \left(1 + \sum_{i=l}^{L-1} \frac{\partial F(x_i, W_i)}{\partial x_i}\right)$$

---

📝 Practice Questions

Q1

<strong>Q1</strong>: Why does the bottleneck block (1×1 → 3×3 → 1×1) reduce computation compared to two 3×3 layers?

For input/output channels = C, bottleneck expansion = 4:

Two 3×3 layers: 2 × (3×3×C×C) = 18C² parameters

Bottleneck: (1×1×C×C/4) + (3×3×C/4×C/4) + (1×1×C/4×C) = C²/4 + 9C²/16 + C²/4 = 4C²/16 + 9C²/16 + 4C²/16 = 17C²/16 ≈ 1.06C²

The bottleneck uses ~6% of the parameters of two 3×3 layers while maintaining representational capacity.

The 1×1 layers reduce dimensionality before the expensive 3×3 convolution, then restore it. Q2

<strong>Q2

<strong>Q2</strong>: In a ResNet-50, the first 1×1 layer in a bottleneck reduces channels from 256 to 64. Why reduce channels?

Reducing channels before the 3×3 conv saves computation:

• Without reduction: 3×3 conv on 256 channels → 3×3×256×256 = 589,824 operations
• With reduction: 1×1 reduces 256→64, then 3×3 on 64 channels → 3×3×64×64 = 36,864 operations
• Total with reduction: 1×1×(256×64) + 3×3×(64×64) + 1×1×(64×256) = 16,384 + 36,864 + 16,384 = 69,632
• Without: 589,824

The bottleneck performs ~8× fewer operations. The receptive field is preserved because the 3×3 conv operates on reduced but spatially meaningful features. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: What happens if you remove skip connections from a 152-layer ResNet and train from scratch?

Without skip connections, the 152-layer network would suffer from:

1. Higher training error: The degradation problem means the deep network can't even fit the training data as well as a shallow one
2. Vanishing gradients: Early layers receive negligible gradient updates
3. Optimization difficulty: The loss landscape becomes much harder to navigate

Empirically, a plain 152-layer CNN would achieve worse training accuracy than ResNet-34, even though it has more parameters and capacity. This is the degradation problem that ResNet was designed to solve.

With skip connections, ResNet-152 uses them to create a highway for gradients and learned residual functions. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: ResNet-18 has 11.7M parameters. If each parameter is FP32 (4 bytes), what's the memory for weights? What about activations for a 224×224 input with batch_size=32?

Weights: 11.7M × 4 bytes = 46.8 MB

Activations (approximate — depends on feature map sizes at each layer):

• After conv1: 112×112×64 = 0.8M per image
• After layer1: 56×56×64 × 2 blocks ≈ 0.4M
• After layer2: 28×28×128 × 2 blocks ≈ 0.2M
• After layer3: 14×14×256 × 2 blocks ≈ 0.1M
• After layer4: 7×7×512 × 2 blocks ≈ 0.05M
• Total activations: ~1.5M per image × batch_size=32 = 48M

In FP32: 48M × 4 bytes = 192 MB for activations

Total memory ≈ 47 MB (weights) + 192 MB (activations) + 94 MB (gradients) + 384 MB (optimizer states for Adam) ≈ 717 MB

This fits on most modern GPUs.

</details> * * * ## 🔗 Cross-References - **Next**: [Object Detection](/notes/04-degree-electives-bsda5006-dl-cv-week04-04-object-detection) - **Video**: BSDA5006 Week 4-5 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**CNN Fundamentals**](/notes/04-degree-electives-bsda5006-dl-cv-week01-01-cnn-fundamentals)[Next**Transfer Learning**](/notes/04-degree-electives-bsda5006-dl-cv-week03-03-transfer-learning)