PyTorch Fundamentals for Deep Learning Practice

🎯 Learning Objectives

• Create and manipulate PyTorch tensors
• Use autograd for automatic differentiation
• Build neural networks with nn.Module
• Implement a complete training loop
• Utilize GPU acceleration

📋 Prerequisites

• Python programming
• Basic understanding of neural networks

---

1. 📖 Core Content

1.1 Tensors: The Core Data Structure

python
Run Inline
import torch
# Creation
x = torch.tensor([1, 2], [3, 4](/courses/bsda5013/notes/1%2C%202%5D%2C%20%5B3%2C%204), dtype=torch.float32)
zeros = torch.zeros(3, 4)
ones = torch.ones(2, 3, 4)
random = torch.randn(5, 10)  # Standard normal
# Operations on GPU
if torch.cuda.is_available():
    x = x.cuda()  # Move to GPU
    print(f"On GPU: {x.device}")
# Reshaping
x_flat = x.view(-1)  # Flatten to 1D
x_flat2 = x.reshape(-1)  # Alternative
x_transposed = x.T  # Transpose
# Indexing (same as NumPy)
print(x[0, :])  # First row
print(x[:, -1])  # Last column
# Broadcasting
a = torch.tensor([1], [2], [3](/courses/bsda5013/notes/1%5D%2C%20%5B2%5D%2C%20%5B3))  # (3, 1)
b = torch.tensor([10, 20, 30])     # (3,) → broadcast to (3, 3)
c = a + b  # (3, 3)

1.2 Autograd: Automatic Differentiation

python
Run Inline
# Gradient tracking
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x[0]**2 + x[1]**3  # 4 + 27 = 31
y.backward()  # Compute gradients
print(f"dy/dx0 = {x.grad[0]:.1f}")  # 2*x[0] = 4
print(f"dy/dx1 = {x.grad[1]:.1f}")  # 3*x[1]^2 = 27
# Gradient accumulation (reset!)
x.grad.zero_()

1.3 Building Models with nn.Module

python
Run Inline
import torch.nn as nn
import torch.nn.functional as F
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(0.2)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.fc3(x)  # No activation on output (will use CrossEntropyLoss)
        return x
model = MLP(784, 256, 10)
print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")

1.4 Training Loop Pattern

python
Run Inline
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
# Hyperparameters
learning_rate = 1e-3
batch_size = 64
num_epochs = 10
# Data
X = torch.randn(1000, 784)
y = torch.randint(0, 10, (1000,))
dataset = TensorDataset(X, y)
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
# Model, Loss, Optimizer
model = MLP(784, 256, 10)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
# Training loop
for epoch in range(num_epochs):
    running_loss = 0.0
    for batch_X, batch_y in dataloader:
        # Forward
        outputs = model(batch_X)
        loss = criterion(outputs, batch_y)
        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    avg_loss = running_loss / len(dataloader)
    print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_loss:.4f}")

1.5 Common Patterns

---

📝 Practice Questions

Q1

<strong>Q1</strong>: Create a tensor of shape (3, 4, 5) filled with random normal values. Permute it to (4, 3, 5).

python
Run Inline
x = torch.randn(3, 4, 5)
x_permuted = x.permute(1, 0, 2)  # (4, 3, 5)
print(x_permuted.shape)  # torch.Size([4, 3, 5])

permute rearranges dimensions. The original dimension 0 (size 3) moves to position 1, original dim 1 (size 4) moves to position 0, and dim 2 stays. Q2

<strong>Q2

<strong>Q2</strong>: Write a training loop that saves the model whenever validation loss reaches a new minimum.

python
Run Inline
best_val_loss = float('inf')

for epoch in range(num_epochs):
    # Training
    model.train()
    for batch in train_loader:
        ...

    # Validation
    model.eval()
    val_loss = 0
    with torch.no_grad():
        for batch in val_loader:
            ...

    # Checkpoint
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'val_loss': val_loss,
        }, 'best_model.pt')
        print(f"Saved new best model (val_loss={val_loss:.4f})")

This implements early stopping with checkpointing — keep the best version of the model, not the last one. Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: Why use torch.no_grad() during evaluation?

torch.no_grad() disables gradient tracking, which:

1. Saves memory: No computation graph stored (can be GBs for large models)
2. Speeds up inference: No backward pass computation needed
3. Prevents accidental updates: Model weights won't change

Without it, evaluating on a test set of 10,000 examples would build a computation graph storing all intermediate activations, using unnecessary memory. Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: What's the difference between model.train() and model.eval()?

model.train():

• Enables dropout (randomly drops neurons)
• Enables batch norm (uses batch statistics)

model.eval():

• Disables dropout (uses all neurons)
• Batch norm uses running averages (not batch statistics)

Forgetting to switch between modes is a common bug: using train() for evaluation gives inconsistent results due to dropout randomness.

</details> * * * ## 🔗 Cross-References - **Next**: [Data Loading & Pipelines](/courses/bsda5013/notes/.%2Fweek02%2F02-data-pipelines) - **Video**: BSDA5013 Week 1-2 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Next**Data Pipelines**](/notes/04-degree-electives-bsda5013-dl-practice-week02-02-data-pipelines-dataloader)