Neural Sync Active

Transfer Learning for Computer Vision

Registry Synced

Transfer Learning for Computer Vision

919 words

5 min read

Reading compass

Now · 🎯 Learning Objectives

Transfer Learning for Computer Vision

🎯 Learning Objectives

Understand why transfer learning works for vision tasks
Implement fine-tuning and feature extraction strategies
Choose appropriate layers to freeze or fine-tune
Apply domain adaptation for custom datasets

📋 Prerequisites

CNN architecture knowledge
Image classification concepts

1. 📖 Core Content

1.1 Why Transfer Learning?

Training CNNs from scratch requires massive datasets (ImageNet: 14M images). For small custom datasets (100-10K images), starting from pre-trained weights is dramatically more effective. Key insight: Lower layers of CNNs learn general features (edges, textures, shapes) that transfer across tasks. Higher layers learn task-specific features.


python
import torchvision.models as models
import torch.nn as nn
# Load pre-trained ResNet-50
model = models.resnet50(pretrained=True)
# Freeze feature extractor
for param in model.parameters():
    param.requires_grad = False
# Replace classifier for new task (10 classes instead of 1000)
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 10)

1.2 Transfer Learning Strategies

Strategy	Dataset Size	Similarity	Approach
Feature Extraction	Small	High	Freeze backbone, train new classifier
Fine-tuning	Medium	High	Train all layers with low LR
Partial Fine-tuning	Small-Medium	Low	Freeze early layers, fine-tune later
Progressive Unfreezing	Medium	Medium	Gradually unfreeze layers from top
From Scratch	Large	Low	Don't use transfer learning

1.3 Code Implementation


python
# Strategy 1: Feature extraction
model = models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
model.fc = nn.Linear(512, num_classes)
# Strategy 2: Full fine-tuning
model = models.resnet50(pretrained=True)
model.fc = nn.Linear(2048, num_classes)
# Train with lower learning rate (e.g., 1e-4 instead of 1e-3)
# Strategy 3: Partial fine-tuning (last 2 layers only)
model = models.resnet50(pretrained=True)
for param in list(model.parameters())[:-4]:  # Freeze all but last layers
    param.requires_grad = False
model.fc = nn.Linear(2048, num_classes)

📝 Practice Questions

Q1

Q1

Q1

Q1: With 100 training images of a rare bird species, would you fine-tune the entire ResNet-50 or just the classifier?

With only 100 images, just train the classifier (feature extraction):

Reasons:

Overfitting risk: 100 images can't train 25M parameters (full ResNet-50). The model will memorize rather than generalize.

Low-level features transfer: Edges, textures, and shapes learned on ImageNet are useful for bird classification.

Computational efficiency: Only training the classifier (2K parameters) is fast and needs little data.

If you must fine-tune: only fine-tune the last 1-2 layers with strong regularization (weight decay, dropout, data augmentation). Q2

Q2

Q2

Q2

Q2

Q2: A pre-trained ImageNet model achieves 95% on a medical X-ray classification task. Is this trustable?

Probably not! The 95% accuracy may be misleading because:

Domain shift: ImageNet features (edges of dogs/cars) may not match X-ray features (bones, anomalies)

Dataset bias: Medical datasets often have strong class imbalance (most X-rays are normal)

Spurious correlations: Model might use hospital markings or scanner artifacts rather than medical features

Accuracy is not calibration: 95% accuracy doesn't mean 95% of positive predictions are correct

Solutions:

Use domain-specific pre-training (e.g., RadImageNet for medical)

Apply grad-CAM to check what the model looks at

Evaluate with balanced metrics (F1, AUC-ROC, precision-recall)

Use domain adaptation techniques to align distributions Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3

Q3: What learning rate should be used for fine-tuning compared to training from scratch?

Lower: Typically 1/10th of the from-scratch learning rate.

Reason: Pre-trained weights are already good. A large learning rate can "destroy" the useful features learned from ImageNet. A lower learning rate makes small, careful adjustments to adapt to the new task.

Typical values:

From scratch: LR = 1e-3 (Adam) or 1e-1 (SGD)

Feature extraction: LR = 1e-3 (same as scratch, only trains classifier)

Full fine-tuning: LR = 1e-4 to 1e-5 (much lower)

Differential learning rates: Early layers (general features) → lowest LR (1e-5). Later layers (task-specific) → higher LR (1e-4). Classifier → highest LR (1e-3).

This differential approach preserves general features while adapting task-specific features. Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4

Q4: What is catastrophic forgetting in fine-tuning, and how can it be prevented?

Catastrophic forgetting: When fine-tuning on a new task, the model's performance on the original task (ImageNet classification) drops dramatically. The model "forgets" the original task.

Why it happens: Gradient updates that improve the new task may damage weights important for the original task. Since we only train on new data, there's no signal to preserve original knowledge.

Prevention:

Lower learning rate: Smaller updates minimize damage

Freeze early layers: Only fine-tune task-specific layers

Elastic Weight Consolidation (EWC): Penalize changes to important parameters

Learning without Forgetting (LwF): Include original task loss during fine-tuning

Progressive networks: Add new columns for new tasks while freezing old ones

Replay: Mix original task data during fine-tuning (e.g., 10% original, 90% new)

For most practical transfer learning, simply freezing early layers and using a low LR is sufficient.

</details> * * * ## 🔗 Cross-References - **Next**: [Object Detection](/notes/04-degree-electives-bsda5006-dl-cv-week04-04-object-detection) - **Previous**: [Advanced CNNs](../week02/02-advanced-cnns.md) - **Video**: BSDA5006 Week 3 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**ResNet & Skip Connections**](/notes/04-degree-electives-bsda5006-dl-cv-week02-02-resnet-skip-connections)[Next**Object Detection**](/notes/04-degree-electives-bsda5006-dl-cv-week04-04-object-detection)