Denoising Diffusion Implicit Models: Accelerated Sampling

🎯 Learning Objectives

• Understand the limitations of DDPM sampling speed and how DDIM addresses them
• Derive the non-Markovian forward process that makes DDIM work
• Implement accelerated sampling with arbitrary step sizes
• Use DDIM for deterministic encoding (inversion) of real images
• Compare DDIM's latent space properties with DDPM

📋 Prerequisites

• Diffusion Models (DDPM) (Week 4): Forward/reverse processes, noise scheduling
• Probability basics: Conditional distributions, Markov chains
• Linear algebra: Matrix operations, eigendecomposition

1. 📖 Core Content

1.1 Intuition: Why DDIM?

• Slow: A single 1024×1024 image can take 10-60 seconds on a high-end GPU
• Impractical for real-time applications: Chat-based image generation, interactive editing
• Energy inefficient: 1000 forward passes through a large neural network DDIM asks: Do we really need all 1000 steps? Can we skip most of them? The key insight: DDPM's forward process is Markovian (each step depends only on the previous step). But the reverse process doesn't need to be Markovian too! By designing a non-Markovian forward process that has the same marginal distributions as DDPM, we can use a shortened reverse sampling path — e.g., 50 steps instead of 1000. Even more surprisingly, DDIM makes the reverse process deterministic (no random noise during sampling), which means:

1. The mapping from noise $z$ to image $x_0$ is one-to-one (injective)
2. We can invert the process: encode a real image back into its latent noise vector
3. Latent space becomes smooth for interpolation

1.2 Review: DDPM's Markovian Forward Process

1.3 The DDIM Non-Markovian Forward Process

1.4 The DDIM Sampling Equation

1.5 Accelerated Sampling: Skipping Steps

Worked Example 1: DDIM Sampling with 10 Steps

Worked Example 2: Comparing DDPM (1000 steps) vs DDIM (50 steps) Quality

1.6 Deterministic Inversion (Encoding)

Why Inversion Matters

1. Image editing: Encode an image, edit the latent, decode back
2. Style mixing: Combine latents from different images
3. Interpolation: Smoothly morph between two images by interpolating their latents

1.7 DDIM vs DDPM: Comparison

1.8 Edge Cases & Gotchas

• Very few steps (1-5): DDIM quality degrades significantly. At 1 step, it just denoises once but doesn't resolve details well.
• Optimal step count: Empirically, 20-50 DDIM steps give comparable quality to 1000 DDPM steps for most datasets.
• Noise scheduling matters: The original schedule (linear, cosine, sigmoid) affects DDIM performance. Cosine schedules generally work better with DDIM.
• Deterministic inversion accumulates error: Long inversion chains (e.g., 1000 steps) can drift from the true path.
• CFG (Classifier-Free Guidance) interaction: DDIM's determinism breaks with CFG scaling > 1, but accelerated sampling still works.

1.9 Why This Matters

• Stable Diffusion uses DDIM schedulers for fast sampling (typically 20-50 steps)
• Image-to-image translation relies on DDIM inversion
• Latent space manipulation (e.g., finding directions for specific attributes) works because DDIM provides a smooth latent space
• Video generation benefits from DDIM's consistency across frames when using shared noise

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Using DDPM Training but DDIM Sampling Without Adjusting the Noise Schedule

Pitfall 2: Expecting DDIM to Replace All DDPM Sampling

Pitfall 3: Assuming DDIM Inversion Is Perfect

4. 📝 Practice Questions

Q1: If DDPM requires 1000 steps and DDIM achieves comparable quality with 50 steps, what's the speedup factor?

The speedup appears to be $1000/50 = 20\times$ in terms of neural network evaluations. However, DDIM's per-step computation is identical to DDPM's (both evaluate $\epsilon_\theta$ once per step), so the actual speedup is indeed $20\times$.

In practice, the speedup may be slightly less due to overhead (data transfers between CPU/GPU), but it's approximately $20\times$ faster. Q2: Show that when $\sigma_t = \sqrt{\tilde{\beta}_t}$, the DDIM general process reduces to DDPM.

The general DDIM reverse distribution is:

$q_\sigma(x_{t-1} | x_t, x_0) = \mathcal{N}\left(x_{t-1}; \sqrt{\bar{\alpha}_{t-1}} x_0 + \sqrt{1-\bar{\alpha}_{t-1} - \sigma_t^2} \cdot \frac{x_t - \sqrt{\bar{\alpha}_t} x_0}{\sqrt{1-\bar{\alpha}_t}}, \sigma_t^2 I\right)$

When $\sigma_t^2 = \tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \beta_t$:

The mean becomes:

$\mu = \sqrt{\bar{\alpha}_{t-1}} x_0 + \sqrt{1-\bar{\alpha}_{t-1} - \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \beta_t} \cdot \frac{x_t - \sqrt{\bar{\alpha}_t} x_0}{\sqrt{1-\bar{\alpha}_t}}$

Using $\beta_t = 1 - \alpha_t = 1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}$:

After simplification (shown in DDPM derivation), this becomes:

$\tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar{\alpha}_{t-1}} \beta_t}{1-\bar{\alpha}_t} x_0 + \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t} x_t$

which is exactly the DDPM posterior mean. And the variance $\sigma_t^2 I = \tilde{\beta}_t I$ is DDPM's posterior variance. Q3: For DDIM, if you start from the same noise vector $x_T$ and run sampling twice, do you get the same image?

Yes — DDIM is deterministic. Given the same $x_T$ and the same trained model $\epsilon_\theta$, the sequence of $x_{T-1}, x_{T-2}, ..., x_0$ is fully determined. There's no stochastic term in the reverse step.

This is fundamentally different from DDPM, where each sample path has random noise injections and would produce different images from the same starting $x_T$.

This determinism is what enables DDIM inversion and interpolation applications. Q4: You want to generate 16 images for a storyboard. You have a batch of 16 noise vectors. With DDPM you need 16 × 1000 = 16000 model evaluations. How many with DDIM using 50 steps?

16 × 50 = 800 model evaluations. That's a 20× reduction.

However, note that modern implementations process the entire batch in parallel (batch size = 16), so it's actually 50 batch evaluations vs 1000 batch evaluations. The wall-clock speedup is still approximately 20×. Q5: Why can't DDPM be used for deterministic image encoding?

DDPM's reverse process includes a stochastic noise term at each step:

$x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t)\right) + \sigma_t z$

Even if we know every $\epsilon_\theta(x_t, t)$ exactly, the random $z \sim \mathcal{N}(0, I)$ at each step means:

1. Starting from the same $x_0$, the forward process gives a distribution over $x_T$ (not a unique $x_T$)
2. Starting from $x_T$, the reverse gives a distribution over $x_0$

There's no deterministic mapping between $x_0$ and $x_T$ in either direction. DDIM removes the stochasticity, creating a bijection. Q6: For DDIM with $T=1000$ and $S=20$, propose a sampling schedule $\tau$ and explain why you chose it.

A common choice is:

$\tau_i = \left\lfloor \frac{i^2}{S^2} \cdot T \right\rfloor \quad \text{for } i = 1, ..., S$

This is a quadratic schedule (compared to uniform). The reasoning:

• Early steps (high noise, near $T$): Larger changes happen, so we want more steps when noise is high
• Late steps (low noise, near 0): Finer details are adjusted, but changes are smaller, so we can take larger jumps

With $S=20$, the schedule might be: $\tau = [1, 3, 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, 120, 136, 153, 171, 190, 210, 231, 253, 276, 300, 325, 351, 378, 406, 435, 465, 496, 528, 561, 595, 630, 666, 703, 741, 780, 820, 861, 903, 946, 990, 1000]$

Wait, that's more than 20. Let me adjust: the quadratic spacing places more steps at the end.

Actually, a simpler and commonly used approach is linear spacing:

$\tau_i = \left\lfloor \frac{i}{S} \cdot T \right\rfloor \text{ for } i = 1, ..., S$

With $S=20$, $T=1000$: $\tau = [50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000]$.

This is simple and works well in practice. Q7: Explain the relationship between DDIM and neural ODEs.

DDIM's deterministic reverse process can be viewed as an Euler discretization of a neural ordinary differential equation (ODE).

In the continuous-time limit (as the number of steps $T \to \infty$), DDIM's iterative equation becomes:

$\frac{dx}{dt} = f_\theta(x, t) \cdot \nabla_x \log p_t(x)$

where $f_\theta$ is derived from the noise prediction network.

This connection is important because:

1. It justifies using ODE solvers (RK45, DOPRI) for even faster sampling
2. It enables likelihood computation via the probability flow ODE
3. It connects diffusion models to score-based generative modeling and normalizing flows

The DDIM ODE perspective is explored further in the Score-Based Generative Modeling literature (Song et al., 2021). Q8: A student claims "DDIM is just DDPM with fewer training steps." Is this correct? Why or why not?

Not correct. DDIM and DDPM have different claims:

• DDPM training: Trains $\epsilon_\theta$ on $T$ timesteps with a Markovian forward process
• DDIM sampling: Uses the same trained model $\epsilon_\theta$ but with a different (non-Markovian) reverse process

DDIM does not change training at all. It's purely a sampling-time modification. The model is still trained with the DDPM objective:

$\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon} \left[ \|\epsilon - \epsilon_\theta(x_t, t)\|^2 \right]$

"Fewer training steps" refers to something different (training the model for fewer iterations), which would make the model worse regardless of the sampling method. Q9: You have a real image $x_0$ and you run DDIM inversion to get $x_T$. Then you run DDIM sampling from $x_T$ back to $x_0'$. Under what conditions would $x_0 = x_0'$?

Perfect reconstruction ($x_0 = x_0'$) would require:

1. Perfect denoiser: $\epsilon_\theta(x_t, t) = \epsilon$ for all $t$, where $\epsilon$ is the actual noise added at step $t$
2. Consistent inversion path: The inversion path from $x_0$ to $x_T$ must use the same steps as the generation path from $x_T$ to $x_0$
3. No numerical error: Floating-point precision doesn't cause drift

In practice, none of these hold perfectly:

• $\epsilon_\theta$ is an imperfect estimator
• There's discretization error from the finite step count
• Floating-point errors accumulate

However, DDIM reconstruction is remarkably good — with 1000 steps, pixel-wise reconstruction error is typically < 5% for natural images, and with 10000 steps it's nearly perfect (for models that can handle the extended schedule). Q10: Design an experiment to verify DDIM's latent space smoothness.

Experiment: Image Interpolation in DDIM Latent Space

Step 1: Take two real images $x_0^{(A)}$ and $x_0^{(B)}$.

Step 2: Encode both to latents $x_T^{(A)}$ and $x_T^{(B)}$ using DDIM inversion.

Step 3: For $\alpha$ in $\{0, 0.1, 0.2, ..., 1.0\}$, compute interpolated latent:

$x_T^{(\alpha)} = \alpha \cdot x_T^{(A)} + (1-\alpha) \cdot x_T^{(B)}$

Step 4: Decode each $x_T^{(\alpha)}$ with DDIM sampling to get $x_0^{(\alpha)}$.

Expected result: For $\alpha = 0$, $x_0^{(0)} \approx x_0^{(B)}$. For $\alpha = 1$, $x_0^{(1)} \approx x_0^{(A)}$. For intermediate $\alpha$, $x_0^{(\alpha)}$ should be a smooth semantic blend — e.g., if $A$ is a photo of a cat and $B$ is a photo of a dog, intermediate values morph continuously between cat and dog.

This would not work with DDPM latents because the stochastic reverse process destroys the linear interpolation structure. Q11: With DDIM, explain why using 1 sampling step ($S=1$) gives a poor result.

With $S=1$, $\tau = [1000]$, so we jump directly from $t=1000$ to $t=0$:

$x_0 = \sqrt{\bar{\alpha}_0} \cdot \frac{x_T - \sqrt{1-\bar{\alpha}_T} \epsilon_\theta(x_T, T)}{\sqrt{\bar{\alpha}_T}} + \sqrt{1-\bar{\alpha}_0} \epsilon_\theta(x_T, T)$

Since $\bar{\alpha}_0 = 1$ and $\bar{\alpha}_T \approx 0$:

$x_0 \approx 1 \cdot \frac{x_T - 1 \cdot \epsilon_\theta(x_T, T)}{0} + 0 \cdot \epsilon_\theta(x_T, T)$

This is problematic because $\sqrt{\bar{\alpha}_T} \approx 0$ causes division by near-zero. In practice, implementations clamp $\bar{\alpha}_t$ to avoid this, but the single-step reconstruction is just $\hat{x}_0 = \frac{x_T}{\sqrt{\bar{\alpha}_T}} \approx$ a scaled noise vector, which contains no meaningful structure.

A single denoising step can only remove a fixed amount of noise — it can't separate structure from noise in one shot. Multiple steps are needed for the iterative refinement that diffusion models rely on. Q12: Suppose you want to use DDIM for video generation where temporal consistency is crucial. How would DDIM's properties help?

For video generation, we want consecutive frames to be consistent (no flickering). DDIM helps because:

1. Shared noise structure: Generate a base latent $x_T$ for the first frame, then for subsequent frames, use small perturbations of $x_T$. Since DDIM deterministically maps latent → image, small latent changes → small image changes.
2. Interpolation between frames: Generate latents for keyframes, then linearly interpolate intermediate latents. DDIM's smooth latent space ensures smooth visual transitions.
3. Consistent inversion: If editing a video, invert each frame independently with DDIM, edit the latents consistently, and decode. The edits will be temporally coherent.

This is why most diffusion-based video models (e.g., Stable Video Diffusion) build on DDIM-like schedulers.

5. 🔗 Cross-References

• Previous: Diffusion Models (DDPM) — Foundation for DDIM
• Next: Autoregressive Models — Alternative generative paradigm
• Related: Conditional Generation — Using DDIM with guidance
• External: Song et al., "Denoising Diffusion Implicit Models" (ICLR 2021) Join Discord PreviousDiffusion Models (DDPM)NextAutoregressive Models