Diffusion Models: DDPM, Forward Process, and Reverse Denoising

🎯 Learning Objectives

• Understand the forward diffusion process as a Markov chain
• Derive the reverse denoising objective
• Implement the noise scheduling and training loss
• Explain the connection to score matching

📋 Prerequisites

• Markov chains
• KL divergence and ELBO (from VAE)
• U-Net architecture

---

1. 📖 Core Content

1.1 Intuition

1. Forward process: Gradually add Gaussian noise to an image until it becomes pure noise
2. Reverse process: Learn to denoise step by step, recovering the original image This is like taking a Polaroid photo and learning how to reverse the development process.

1.2 Forward Diffusion Process

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$$ $$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

1.3 Reverse Process

$$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$$ $$\mathcal{L}_{simple} = \mathbb{E}_{t, x_0, \epsilon} [\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$$

python
Run Inline
# runnable
import torch
import torch.nn as nn
import numpy as np
class DDPM(nn.Module):
    """Denoising Diffusion Probabilistic Model"""
    def __init__(self, model, T=1000, beta_start=1e-4, beta_end=0.02):
        super().__init__()
        self.model = model  # U-Net or similar
        self.T = T
        # Noise schedule (linear)
        self.betas = torch.linspace(beta_start, beta_end, T)
        self.alphas = 1 - self.betas
        self.alpha_bars = torch.cumprod(self.alphas, dim=0)
    def forward_process(self, x0, t):
        """Add noise to reach timestep t: x_t = √ᾱ·x0 + √(1-ᾱ)·ε"""
        sqrt_alpha_bar = torch.sqrt(self.alpha_bars[t])[:, None, None, None]
        sqrt_one_minus = torch.sqrt(1 - self.alpha_bars[t])[:, None, None, None]
        noise = torch.randn_like(x0)
        xt = sqrt_alpha_bar * x0 + sqrt_one_minus * noise
        return xt, noise
    def training_step(self, x0):
        """Sample t, add noise, predict noise"""
        t = torch.randint(0, self.T, (x0.shape[0],))
        xt, noise = self.forward_process(x0, t)
        # Predict noise
        noise_pred = self.model(xt, t)
        # Simple loss
        loss = nn.MSELoss()(noise_pred, noise)
        return loss
    @torch.no_grad()
    def sample(self, n_samples, img_shape, device):
        """Generate new samples by reversing diffusion"""
        x = torch.randn(n_samples, *img_shape).to(device)
        for t in reversed(range(self.T)):
            # Predict noise
            noise_pred = self.model(x, torch.full((n_samples,), t).to(device))
            # Compute x_{t-1}
            alpha = self.alphas[t]
            alpha_bar = self.alpha_bars[t]
            # Coefficient for predicted x0 contribution
            coef1 = 1 / torch.sqrt(alpha)
            coef2 = (1 - alpha) / torch.sqrt(1 - alpha_bar)
            x = coef1 * (x - coef2 * noise_pred)
            # Add noise (except at t=0)
            if t > 0:
                noise = torch.randn_like(x) * torch.sqrt(self.betas[t])
                x = x + noise
        return x

1.4 Noise Schedule

1.5 U-Net Architecture

• Encoder (downsampling) + Decoder (upsampling) with skip connections
• Self-attention layers at lower resolutions
• Time embedding (sinusoidal) injected at each layer

1.6 Connection to Score Matching

$$\epsilon_\theta(x_t, t) = -\sqrt{1 - \bar{\alpha}_t} \nabla_{x_t} \log p(x_t)$$

---

📝 Practice Questions

Q1

<strong>Q1</strong>: In DDPM, why can we sample x_t directly from x_0 without iterating through all intermediate steps?

The forward process adds Gaussian noise at each step: $x_t = \sqrt{1-\beta_t} x_{t-1} + \sqrt{\beta_t} \epsilon_t$

By the properties of Gaussian distributions, adding noise sequentially is equivalent to adding a single Gaussian with accumulated variance: $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$

Where $\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)$. This is because the sum of independent Gaussian noises is still Gaussian, with variance equal to the cumulative product.

This closed form enables efficient training: we can sample any t uniformly and compute the loss, rather than rolling out the full chain. Q2

<strong>Q2

<strong>Q2</strong>: The simple loss L_simple doesn't contain a KL term like VAE. Why not?

The full DDPM loss derived from ELBO includes KL terms at each timestep. However, when the forward process variances are fixed (not learned), the KL term simplifies to a comparison between the true denoising distribution and the model's prediction.

With the parameterization predicting noise $\epsilon$ instead of the mean $\mu$, the loss simplifies to: $\|\epsilon - \epsilon_\theta(x_t, t)\|^2$

This is equivalent to a reweighted ELBO that emphasizes difficult denoising steps (middle t values) and de-emphasizes easy steps (t≈0 and t≈T). Empirically, this reweighting produces better samples than the full ELBO.

So L_simple is a modified ELBO, not a different objective. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: Why does DDPM typically need 1000 sampling steps, making generation slow?

DDPM's reverse process is a Markov chain with T=1000 steps. Each step requires one forward pass through the U-Net. Generating a single 256×256 image requires 1000 U-Net evaluations.

Why 1000 steps? Because the forward process adds small amounts of noise per step (β_t ≈ 10^{-4} to 0.02). Each reverse step removes only a tiny amount of noise. Using fewer steps would require larger denoising jumps, which the model wasn't trained for.

Solutions:

• DDIM: Deterministic sampling, can use 50-100 steps (Week 5)
• DPM-Solver: ODE-based solver, 10-25 steps
• LCM (Latent Consistency Models): 1-4 steps

These trade generation speed for a small quality reduction. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: In DDPM training, t is sampled uniformly from [0, T-1]. Why is this important?

Uniform sampling of t ensures:

1. All noise levels are covered: The model learns to denoise at all levels of corruption
2. No bias: If t were concentrated at the middle, the model would be bad at near-clean (t≈0) and near-noise (t≈T) denoising
3. Each step gets equal training signal: The loss for each t is weighted equally

During generation, the model needs to do all T steps well. A single bad step early in the chain can corrupt the entire sample. Uniform training ensures uniform competence.

In practice, some implementations use a different distribution over t to emphasize harder (middle t) steps.

</details> * * * ## 🔗 Cross-References - **Next**: [DDIM](/notes/04-degree-electives-bsda5002-genai-foundations-week05-05-ddim) - **Previous**: [VAEs](/notes/04-degree-electives-bsda5002-genai-foundations-week03-03-vaes) - **Video**: BSDA5002 Week 6-7 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Variational Autoencoders**](/notes/04-degree-electives-bsda5002-genai-foundations-week03-03-vaes)[Next**DDIMs**](/notes/04-degree-electives-bsda5002-genai-foundations-week05-05-ddim)