Semantic and Instance Segmentation: FCN, U-Net, Mask R-CNN

🎯 Learning Objectives

• Distinguish semantic, instance, and panoptic segmentation
• Implement a Fully Convolutional Network (FCN)
• Understand U-Net's encoder-decoder with skip connections
• Explain Mask R-CNN's multi-task learning

📋 Prerequisites

• CNN fundamentals
• Object detection concepts
• Image classification

---

1. 📖 Core Content

1.1 Types of Segmentation

1.2 Fully Convolutional Networks (FCN)

python
Run Inline
class FCN(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        # Encoder (backbone)
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.conv2 = nn.Conv2d(64, 128, 3, padding=1)
        self.pool = nn.MaxPool2d(2)
        # Decoder (upsampling)
        self.upconv1 = nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1)
        self.upconv2 = nn.ConvTranspose2d(64, num_classes, 4, stride=2, padding=1)
    def forward(self, x):
        # Encoder
        x = F.relu(self.conv1(x))  # (N, 64, H, W)
        x = self.pool(x)            # (N, 64, H/2, W/2)
        x = F.relu(self.conv2(x))   # (N, 128, H/2, W/2)
        x = self.pool(x)            # (N, 128, H/4, W/4)
        # Decoder
        x = F.relu(self.upconv1(x)) # (N, 64, H/2, W/2)
        x = self.upconv2(x)          # (N, C, H, W) — original resolution
        return x  # No softmax (use CrossEntropyLoss)

1.3 U-Net Architecture

1.4 Loss Functions

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1</strong>: For a 256×256 input image with 3 channels, how does U-Net handle the spatial dimension changes?

Encoder path:

• Input: 256×256×3
• After first conv block: 256×256×64
• After first pooling: 128×128×64
• After second conv block: 128×128×128
• After second pooling: 64×64×128
• Bottleneck: 64×64×256

Decoder path:

• First upconv: 128×128×128 (concatenated with encoder features of same size)
• Second upconv: 256×256×64
• Final 1×1 conv: 256×256×num_classes

Output has same spatial dimensions as input (256×256) — perfect for segmentation. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Why does Mask R-CNN output three things: class label, bounding box, and segmentation mask?

Mask R-CNN extends Faster R-CNN with a mask branch. It solves three related tasks simultaneously:

1. Classification: What object is it?
2. Bounding box regression: Where is it?
3. Segmentation mask: What's its shape?

The multi-task loss: L = L_class + L_box + L_mask

Joint training improves all three tasks — the mask branch provides additional supervision that helps classification and localization, and vice versa.

The mask branch uses a small FCN applied to each RoI, predicting a binary mask for each class independently (per-pixel sigmoid). This avoids competition between classes. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: In semantic segmentation, why is Dice loss preferred over cross-entropy for medical images?

Medical images often have severe class imbalance: tumor pixels might be <1% of total pixels.

Cross-entropy: Optimized per-pixel. If 99% of pixels are "background," the model can achieve 99% accuracy by predicting "background" for everything — useless for detecting tumors.

Dice loss: Measures overlap between prediction and ground truth. A perfect background prediction with zero tumor overlap gives Dice = 0 (max loss = 1). This forces the model to actually predict the rare class correctly.

The Dice coefficient is F1-score for segmentation: Dice = 2|A∩B|/(|A|+|B|). Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Instance segmentation is harder than semantic segmentation. Why?

Instance segmentation must:

1. Separate objects: Two adjacent cars of the same class must have different instance IDs
2. Variable number of outputs: Different images have different numbers of objects
3. Overlapping objects: Objects can occlude each other

Semantic segmentation only assigns a class to each pixel — adjacent same-class pixels are treated as one.

Instance segmentation approaches:

• Detection-based (Mask R-CNN): Detect first, then segment
• Grouping-based: Predict embeddings, cluster pixels into instances
• Transformer-based (MaskFormer, DETR): Unified architecture for both

</details> * * * ## 🔗 Cross-References - **Next**: [GANs](/notes/04-degree-electives-bsda5006-dl-cv-week06-06-gans) - **Previous**: [Object Detection](/notes/04-degree-electives-bsda5006-dl-cv-week04-04-object-detection) - **Video**: BSDA5006 Week 7 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Object Detection**](/notes/04-degree-electives-bsda5006-dl-cv-week04-04-object-detection)[Next**GANs for CV**](/notes/04-degree-electives-bsda5006-dl-cv-week06-06-gans)