Generative Adversarial Networks: Theory and Implementation

🎯 Learning Objectives

• Understand the min-max game formulation of GANs
• Implement a GAN with PyTorch
• Diagnose and fix training instability and mode collapse
• Compare GAN variants (DCGAN, WGAN, Conditional GAN)

📋 Prerequisites

• Neural network fundamentals
• PyTorch basics
• Probability distributions

---

1. 📖 Core Content

1.1 The GAN Framework

• Generator G(z): Maps random noise z to fake data x_fake
• Discriminator D(x): Predicts whether x is real or fake Generator goal: Fool the discriminator (make D(G(z)) → 1) Discriminator goal: Distinguish real from fake (D(x_real) → 1, D(x_fake) → 0)

1.2 GAN Loss Function

$$V(D, G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$$

1.3 Training GANs

python
Run Inline
# Generator
class Generator(nn.Module):
    def __init__(self, latent_dim=100, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.ReLU(),
            nn.BatchNorm1d(256),
            nn.Linear(256, 512),
            nn.ReLU(),
            nn.BatchNorm1d(512),
            nn.Linear(512, img_dim),
            nn.Tanh()  # Output in [-1, 1]
        )
    def forward(self, z):
        return self.net(z)
# Discriminator
class Discriminator(nn.Module):
    def __init__(self, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 128),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(128, 1),
            nn.Sigmoid()  # Output probability
        )
    def forward(self, x):
        return self.net(x)

1.4 Training Loop

python
Run Inline
# Initialize
generator = Generator(latent_dim=100)
discriminator = Discriminator()
criterion = nn.BCELoss()
g_optim = optim.Adam(generator.parameters(), lr=2e-4, betas=(0.5, 0.999))
d_optim = optim.Adam(discriminator.parameters(), lr=2e-4, betas=(0.5, 0.999))
# Training
for epoch in range(num_epochs):
    for real_imgs, _ in dataloader:
        batch_size = real_imgs.size(0)
        # === Train Discriminator ===
        # Real images
        d_optim.zero_grad()
        real_pred = discriminator(real_imgs)
        d_real_loss = criterion(real_pred, torch.ones_like(real_pred))
        # Fake images
        z = torch.randn(batch_size, 100)
        fake_imgs = generator(z)
        fake_pred = discriminator(fake_imgs.detach())  # detach to not train generator
        d_fake_loss = criterion(fake_pred, torch.zeros_like(fake_pred))
        d_loss = d_real_loss + d_fake_loss
        d_loss.backward()
        d_optim.step()
        # === Train Generator ===
        g_optim.zero_grad()
        z = torch.randn(batch_size, 100)
        fake_imgs = generator(z)
        fake_pred = discriminator(fake_imgs)
        g_loss = criterion(fake_pred, torch.ones_like(fake_pred))  # Want D(G(z)) → 1
        g_loss.backward()
        g_optim.step()

1.5 Common GAN Problems

1.6 GAN Variants

---

📝 Practice Questions

Q1

<strong>Q1</strong>: At Nash equilibrium in a GAN, what is the optimal discriminator output for real and fake samples?

At equilibrium: $D(x) = \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}$ for any sample x.

When the generator perfectly matches the real data distribution ($p_g = p_{data}$):

• D(x) = 0.5 for both real and fake samples
• The discriminator can't distinguish real from fake (random guessing)

This is the theoretical optimum — the discriminator is maximally confused. Q2

<strong>Q2

<strong>Q2</strong>: Why does the original GAN loss cause vanishing gradients when the discriminator is too strong?

When D is too strong: D(G(z)) ≈ 0 for fake images (D correctly identifies them as fake).

Generator loss: $\log(1 - D(G(z))) \approx \log(1) = 0$

The gradient of $\log(1 - D(G(z)))$ is very small when D(G(z)) is close to 0. The generator gets no useful gradient signal and stops learning.

Fix: Use "−log D trick" — train generator to maximize $\log D(G(z))$ instead of minimizing $\log(1 - D(G(z)))$. This provides stronger gradients when D is winning. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: In WGAN, what does the Wasserstein distance provide over the original GAN loss?

The Wasserstein-1 distance (Earth Mover's distance) provides:

1. Meaningful loss: Decreases as samples improve (original GAN loss doesn't correlate with sample quality)
2. Stable gradients: Doesn't saturate, even when D is strong
3. Convergence signal: Loss correlates with sample quality → can stop training at right time
4. No mode collapse: Theoretically avoids mode collapse

WGAN enforces 1-Lipschitz continuity via weight clipping (WGAN) or gradient penalty (WGAN-GP). Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: A GAN trained on MNIST generates only the digit "1". What's happening and how do you fix it?

This is mode collapse — the generator found a single mode of the distribution that fools the discriminator.

Fixes:

1. WGAN/WGAN-GP: Use Wasserstein loss which penalizes mode collapse
2. Minibatch discrimination: Let D compare samples within a batch to detect lack of diversity
3. Unrolled GANs: Update G using future D parameters
4. Ensemble GANs: Train multiple generators
5. Larger latent space: More noise dimensions may help diversity
6. Add diversity metric to generator loss: Penalize similar outputs

Mode collapse remains an active research challenge, but WGAN-GP with proper hyperparameters can often prevent it.

</details> * * * ## 🔗 Cross-References - **Next**: [VAEs](/notes/04-degree-electives-bsda5002-genai-foundations-week03-03-vaes) - **Video**: BSDA5002 Week 1-3 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Generative Models Overview**](/notes/04-degree-electives-bsda5002-genai-foundations-week01-01-generative-model-overview)[Next**Variational Autoencoders**](/notes/04-degree-electives-bsda5002-genai-foundations-week03-03-vaes)