Neural Sync Active

Vision Transformers (ViT) and Attention for Images

Registry Synced

Vision Transformers (ViT) and Attention for Images

905 words

5 min read

Reading compass

Now · 🎯 Learning Objectives

Vision Transformers (ViT) and Attention for Images

🎯 Learning Objectives

Understand how Transformers are adapted for image data
Implement patch embedding and positional encoding for images
Compare ViT with CNN architectures
Explain hybrid and modern ViT variants

📋 Prerequisites

Transformer architecture (from LLMs)
CNN fundamentals
Self-attention mechanism

1. 📖 Core Content

1.1 From Sequences to Images

Challenge: Transformers process sequences of tokens. Images are 2D grids of pixels. ViT solution: Split image into patches, flatten each patch into a vector, treat patches as a sequence:


python
def patchify(image, patch_size=16):
    """Split image into patches and embed"""
    B, C, H, W = image.shape
    num_patches = (H // patch_size) * (W // patch_size)
    # Reshape: (B, C, H, W) → (B, num_patches, C * patch_size * patch_size)
    patches = image.unfold(2, patch_size, patch_size)\
                  .unfold(3, patch_size, patch_size)\
                  .permute(0, 2, 3, 1, 4, 5)\
                  .reshape(B, num_patches, -1)
    return patches
# 224×224 image, patch_size=16 → 14×14 = 196 patches, each 16×16×3 = 768 dims

1.2 ViT Architecture

(Diagram) Transformer encoder (same as BERT):

Multi-head self-attention (bidirectional)
MLP with GELU activation
Layer Normalization (Pre-LN)
Residual connections

1.3 ViT Variants

Model	Patch Size	Layers	Hidden Dim	Heads	Params	ImageNet Top-1
ViT-B/16	16×16	12	768	12	86M	77.9%
ViT-L/16	16×16	24	1024	16	307M	76.5%
ViT-H/14	14×14	32	1280	16	632M	78.0%
DeiT-B	16×16	12	768	12	86M	81.8%
Swin-B	4×4 (window)	24	1024	16	88M	83.5%

1.4 ViT vs CNN

Aspect	CNN	ViT
Inductive bias	Strong (locality, translation equivariance)	Weak (learned from data)
Data efficiency	Good with small data	Needs large data (≥14M images)
Global context	Limited (needs deep networks)	Natural (attention over all patches)
Parameter efficiency	High (weight sharing)	Lower
Computational cost	Grows linearly with resolution	Grows quadratically with patches
Pre-training	ImageNet (1M) sufficient	JFT-300M (300M, internal Google) or CLIP

📝 Practice Questions

Q1

Q1

Q1

Q1

Q1

Q1: For a 224×224 image with patch_size=16, how many patches are there? What's the sequence length?

Patches per dimension: 224/16 = 14 Total patches: 14×14 = 196 Sequence length (including [CLS]): 197 tokens

This is much shorter than typical NLP sequences (512 tokens for BERT). The quadratic attention cost O(n²) = 197² = 38,809 — very manageable.

For higher resolution (e.g., 512×512 with patch=16): 32×32 = 1024 patches. O(n²) = 1M — still manageable. For 512×512 with patch=8: 4096 patches. O(n²) = 16.7M — getting expensive. This is why ViT uses larger patches for high-resolution inputs. Q2

Q2

Q2

Q2

Q2

Q2

Q2

Q2

Q2: Why does ViT need more pre-training data than CNNs (300M vs 1M images)?

CNNs have strong inductive biases built into the architecture:

Locality: Convolution only looks at local neighborhoods

Translation equivariance: Same feature detector applied everywhere

Hierarchical: Coarse-to-fine feature learning

These biases make CNNs data-efficient — they're pre-wired to process images effectively.

ViT has minimal inductive bias — it's a general sequence model applied to patches. All spatial relationships must be learned from data. Without large pre-training, ViT doesn't learn that nearby patches are related, that edges are meaningful, or that objects are locally coherent.

With enough data (300M images), ViT surpasses CNNs by learning better global relationships. This is why ViT + JFT-300M outperforms ResNet + ImageNet. Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3: How does the Swin Transformer improve upon ViT?

Swin Transformer (Shifted Window) addresses ViT's limitations:

Windowed attention: Compute attention within local windows (7×7 patches) instead of globally. This reduces O(n²) to O(n·w²) where w is window size — linear instead of quadratic in image size.

Shifted windows: Alternate between regular and shifted window partitioning, enabling cross-window connections. This restores global context while maintaining efficiency.

Hierarchical representations: Like CNNs, Swin produces multi-scale features (4×, 8×, 16× downsampling), making it suitable for detection and segmentation.

Relative position bias: Uses relative position within windows instead of absolute position, improving generalization.

Swin achieves CNN-like efficiency with Transformer-like performance, making it practical for high-resolution vision tasks. Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4: In ViT, what does the [CLS] token learn, and how is it used for classification?

The [CLS] token (same idea as BERT) is a learned embedding prepended to the patch sequence. During training:

[CLS] attends to all image patches via self-attention

All patches also attend to [CLS]

The final [CLS] representation aggregates global image information

For classification, a small MLP head is applied to the [CLS] token's output:
python
class_prediction = mlp_head(vit_output[:, 0, :]) # 0 = [CLS] index
Alternatively (used in some modern ViTs): Global average pooling over all patch tokens can replace [CLS], sometimes giving better results.

The [CLS] approach works because self-attention naturally creates a [CLS] representation that incorporates information from all patches — analogous to global average pooling but learned rather than hard-coded.

</details> * * * ## 🔗 Cross-References - **Next**: [Data Augmentation](/notes/04-degree-electives-bsda5006-dl-cv-week08-08-data-augmentation) - **Previous**: [GANs](/notes/04-degree-electives-bsda5006-dl-cv-week06-06-gans) - **Video**: BSDA5006 Week 9 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Data Augmentation**](/notes/04-degree-electives-bsda5006-dl-cv-week08-08-data-augmentation)[Next**Image Processing**](/notes/04-degree-electives-bsda5006-dl-cv-week09-09-image-processing)