Data Loading & Pipelines: Dataset, DataLoader, Transformations

🎯 Learning Objectives

• Create custom Dataset classes
• Use DataLoader with batching, shuffling, and parallelism
• Implement data transformations and augmentation
• Optimize data loading for training throughput

📋 Prerequisites

• PyTorch basics
• Python OOP

---

1. 📖 Core Content

1.1 The Data Loading Problem

1.2 Dataset Class

python
Run Inline
from torch.utils.data import Dataset, DataLoader
import torch
class ImageDataset(Dataset):
    """Custom Dataset for image classification"""
    def __init__(self, file_paths, labels, transform=None):
        self.file_paths = file_paths
        self.labels = labels
        self.transform = transform
    def __len__(self):
        return len(self.file_paths)
    def __getitem__(self, idx):
        # Load image (heavy I/O happens here)
        image = load_image(self.file_paths[idx])  # PIL or numpy
        label = self.labels[idx]
        if self.transform:
            image = self.transform(image)
        return {'image': image, 'label': label}
# Usage
dataset = ImageDataset(file_paths, labels, transform=transforms.ToTensor())

1.3 DataLoader Features

python
Run Inline
dataloader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,        # Important for training
    num_workers=4,       # Parallel data loading
    pin_memory=True,     # Faster GPU transfer
    drop_last=True,      # Drop incomplete batches
    prefetch_factor=2,   # Prefetch batches in advance
    persistent_workers=True  # Keep workers alive between epochs
)

1.4 Performance Comparison

1.5 Transforms

python
Run Inline
from torchvision import transforms
# Training transforms (with augmentation)
train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])
# Validation transforms (no augmentation)
val_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1</strong>: Dataset has 10,000 images, batch_size=64. How many batches per epoch? What if drop_last=False?

Batches = ceil(10000 / 64) = ceil(156.25) = 157 batches with drop_last=False Batches = floor(10000 / 64) = 156 batches with drop_last=True

With drop_last=True, the last incomplete batch (16 images) is dropped. This is important for batch normalization (needs consistent batch statistics) and for model parallelism.

When drop_last=False, the last batch has 16 images, which may cause issues if the model expects consistent batch dimensions. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: With 4 workers, Dataset returns (image_tensor, label). DataLoader with batch_size=32 produces what shape?

Each worker produces individual samples: (3, 224, 224) image + () scalar label.

DataLoader collates them into batches:

• images: (32, 3, 224, 224) — stacked tensor
• labels: (32,) — stacked tensor

If samples have different sizes (e.g., different image dimensions), you need a custom collate_fn to pad or handle variable sizes. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: GPU utilization is 40% during training. How can data loading be optimized?

Low GPU utilization → CPU data loading bottleneck. Solutions:

1. Increase num_workers: From 0 to 4-8 (depends on CPU cores)
2. Enable pin_memory: Faster CPU→GPU transfer
3. Enable prefetch: Load next batch while current batch computes
4. Disable disk I/O: Use SSD instead of HDD, or pre-load data to RAM
5. Simplify transforms: Reduce CPU complexity of transformations
6. Cache preprocessed data: Save transformed data to disk after first epoch
7. Use persistent_workers: Avoid worker restart overhead between epochs

Rule of thumb: GPU utilization > 90% means data pipeline is efficient. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: A custom Dataset loads large CSVs in getitem. Training is slow. What's wrong and how to fix?

Problem: getitem reads from disk for every sample in every epoch. If the CSV has 100K rows and you train for 10 epochs, you read the CSV 1 million times!

Fix: Pre-load the data in init:

python
Run Inline
class BadDataset(Dataset):
    def __getitem__(self, idx):
        # BAD: I/O per sample per epoch
        return pd.read_csv(f"data_{idx}.csv")

class GoodDataset(Dataset):
    def __init__(self):
        # GOOD: Load once in memory
        self.data = [pd.read_csv(f"data_{i}.csv") for i in range(n)]

    def __getitem__(self, idx):
        return self.data[idx]  # Fast RAM access

For datasets too large for RAM: use memory mapping, database indexing, or lazy loading with caching.

</details> * * * ## 🔗 Cross-References - **Next**: [Pandas for DL](/notes/04-degree-electives-bsda5013-dl-practice-week03-03-pandas-dl) - **Previous**: [PyTorch Basics](/notes/04-degree-electives-bsda5013-dl-practice-week01-01-pytorch-basics) - **Video**: BSDA5013 Week 2 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**PyTorch Fundamentals**](/notes/04-degree-electives-bsda5013-dl-practice-week01-01-pytorch-basics)[Next**Pandas for DL**](/notes/04-degree-electives-bsda5013-dl-practice-week03-03-pandas-dl)