Vision Transformers (ViT) and Attention for Images

🎯 Learning Objectives

• Understand how Transformers are adapted for image data
• Implement patch embedding and positional encoding for images
• Compare ViT with CNN architectures
• Explain hybrid and modern ViT variants

📋 Prerequisites

• Transformer architecture (from LLMs)
• CNN fundamentals
• Self-attention mechanism

---

1. 📖 Core Content

1.1 From Sequences to Images

python
Run Inline
def patchify(image, patch_size=16):
    """Split image into patches and embed"""
    B, C, H, W = image.shape
    num_patches = (H // patch_size) * (W // patch_size)
    # Reshape: (B, C, H, W) → (B, num_patches, C * patch_size * patch_size)
    patches = image.unfold(2, patch_size, patch_size)\
                  .unfold(3, patch_size, patch_size)\
                  .permute(0, 2, 3, 1, 4, 5)\
                  .reshape(B, num_patches, -1)
    return patches
# 224×224 image, patch_size=16 → 14×14 = 196 patches, each 16×16×3 = 768 dims

1.2 ViT Architecture

• Multi-head self-attention (bidirectional)
• MLP with GELU activation
• Layer Normalization (Pre-LN)
• Residual connections

1.3 ViT Variants

1.4 ViT vs CNN

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1

<strong>Q1

<strong>Q1

<strong>Q1</strong>: For a 224×224 image with patch_size=16, how many patches are there? What's the sequence length?

Patches per dimension: 224/16 = 14 Total patches: 14×14 = 196 Sequence length (including [CLS]): 197 tokens

This is much shorter than typical NLP sequences (512 tokens for BERT). The quadratic attention cost O(n²) = 197² = 38,809 — very manageable.

For higher resolution (e.g., 512×512 with patch=16): 32×32 = 1024 patches. O(n²) = 1M — still manageable. For 512×512 with patch=8: 4096 patches. O(n²) = 16.7M — getting expensive. This is why ViT uses larger patches for high-resolution inputs. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Why does ViT need more pre-training data than CNNs (300M vs 1M images)?

CNNs have strong inductive biases built into the architecture:

• Locality: Convolution only looks at local neighborhoods
• Translation equivariance: Same feature detector applied everywhere
• Hierarchical: Coarse-to-fine feature learning

These biases make CNNs data-efficient — they're pre-wired to process images effectively.

ViT has minimal inductive bias — it's a general sequence model applied to patches. All spatial relationships must be learned from data. Without large pre-training, ViT doesn't learn that nearby patches are related, that edges are meaningful, or that objects are locally coherent.

With enough data (300M images), ViT surpasses CNNs by learning better global relationships. This is why ViT + JFT-300M outperforms ResNet + ImageNet. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: How does the Swin Transformer improve upon ViT?

Swin Transformer (Shifted Window) addresses ViT's limitations:

1. Windowed attention: Compute attention within local windows (7×7 patches) instead of globally. This reduces O(n²) to O(n·w²) where w is window size — linear instead of quadratic in image size.
2. Shifted windows: Alternate between regular and shifted window partitioning, enabling cross-window connections. This restores global context while maintaining efficiency.
3. Hierarchical representations: Like CNNs, Swin produces multi-scale features (4×, 8×, 16× downsampling), making it suitable for detection and segmentation.
4. Relative position bias: Uses relative position within windows instead of absolute position, improving generalization.

Swin achieves CNN-like efficiency with Transformer-like performance, making it practical for high-resolution vision tasks. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: In ViT, what does the [CLS] token learn, and how is it used for classification?

The [CLS] token (same idea as BERT) is a learned embedding prepended to the patch sequence. During training:

• [CLS] attends to all image patches via self-attention
• All patches also attend to [CLS]
• The final [CLS] representation aggregates global image information

For classification, a small MLP head is applied to the [CLS] token's output:

python
Run Inline
class_prediction = mlp_head(vit_output[:, 0, :])  # 0 = [CLS] index

Alternatively (used in some modern ViTs): Global average pooling over all patch tokens can replace [CLS], sometimes giving better results.

The [CLS] approach works because self-attention naturally creates a [CLS] representation that incorporates information from all patches — analogous to global average pooling but learned rather than hard-coded.

</details> * * * ## 🔗 Cross-References - **Next**: [Data Augmentation](/notes/04-degree-electives-bsda5006-dl-cv-week08-08-data-augmentation) - **Previous**: [GANs](/notes/04-degree-electives-bsda5006-dl-cv-week06-06-gans) - **Video**: BSDA5006 Week 9 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Data Augmentation**](/notes/04-degree-electives-bsda5006-dl-cv-week08-08-data-augmentation)[Next**Image Processing**](/notes/04-degree-electives-bsda5006-dl-cv-week09-09-image-processing)