Conditional Generation and Classifier-Free Guidance

🎯 Learning Objectives

• Understand conditioning in generative models
• Implement classifier guidance for diffusion models
• Explain classifier-free guidance and its advantages
• Apply conditioning to GANs and VAEs

📋 Prerequisites

• GANs, VAEs, and Diffusion models
• Basic understanding of conditional probability

---

1. 📖 Core Content

1.1 The Need for Control

• Class labels (dog, cat, car)
• Text descriptions ("a red car on a beach")
• Images (sketch → photo, night → day)
• Segmentation maps (layout → image)

1.2 Conditioning in Diffusion Models

Classifier Guidance

$$\tilde{\epsilon}_\theta(x_t, t, y) = \epsilon_\theta(x_t, t) - s \cdot \sigma_t \nabla_{x_t} \log p_\theta(y | x_t)$$

• $s = 0$: Unconditional generation
• $s = 1$: Standard conditional
• $s > 1$: Classifier guidance (stronger conditioning)

Classifier-Free Guidance (CFG)

$$\tilde{\epsilon}_\theta = \epsilon_\theta(x_t, t, \emptyset) + w \cdot (\epsilon_\theta(x_t, t, y) - \epsilon_\theta(x_t, t, \emptyset))$$

python
Run Inline
# Classifier-free guidance sampling (simplified)
def sample_with_cfg(model, condition, guidance_scale=3.0, T=1000):
    x = torch.randn(1, 3, 256, 256)
    for t in reversed(range(T)):
        t_tensor = torch.full((1,), t)
        # Unconditional prediction (null condition)
        eps_uncond = model(x, t_tensor, torch.zeros_like(condition))
        # Conditional prediction
        eps_cond = model(x, t_tensor, condition)
        # CFG: guided prediction
        eps = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
        # Denoising step
        x = denoise_step(x, eps, t)
    return x

1.3 Conditioning in GANs

$$G(z, c), D(x, c)$$

python
Run Inline
class ConditionalGenerator(nn.Module):
    def __init__(self, latent_dim=100, num_classes=10):
        super().__init__()
        self.embed = nn.Embedding(num_classes, 50)
        self.net = nn.Sequential(
            nn.Linear(latent_dim + 50, 256),
            nn.ReLU(),
            nn.Linear(256, 784),
            nn.Tanh()
        )
    def forward(self, z, labels):
        c = self.embed(labels)
        x = torch.cat([z, c], dim=1)
        return self.net(x)

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1

<strong>Q1</strong>: In classifier guidance, what happens when the guidance scale s is very large (e.g., 100)?

When s is very large:

1. Strong conditioning: The classifier gradient dominates the noise prediction
2. Image quality drops: The denoising process is pulled too aggressively toward high classifier confidence regions
3. Mode collapse: Generated images converge to a few high-confidence templates
4. Unnatural images: The classifier's gradient may push the image outside the natural image manifold

Typical guidance scales: 1-10 for good quality. Very high s produces images that are clearly of the requested class but look unnatural.

Classifier-free guidance has similar behavior — very high guidance weights (>10) produce over-saturated, unnatural images. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Why does CFG train with both conditional and unconditional objectives?

CFG needs both conditional $\epsilon_\theta(x_t, t, y)$ and unconditional $\epsilon_\theta(x_t, t, \emptyset)$ predictions to compute the guidance vector:

$\epsilon_{guided} = \epsilon_{uncond} + w \cdot (\epsilon_{cond} - \epsilon_{uncond})$

Training jointly with both objectives:

• 90% of batches: Train with condition (class label, text embedding)
• 10% of batches: Train with null condition ($\emptyset$) — randomly drop condition

This ensures the model learns both conditional and unconditional denoising. At inference, the difference $\epsilon_{cond} - \epsilon_{uncond}$ provides the "direction" toward the condition, and $w$ controls how strongly to follow that direction.

The same model handles both cases, making CFG efficient and practical. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3</strong>: How does text-to-image generation (like Stable Diffusion) connect text conditioning with image generation?

Stable Diffusion uses a text encoder (CLIP or T5) to convert text prompts into embeddings, then conditions the diffusion model on these embeddings:

1. Text encoding: "a red car" → CLIP text encoder → text embedding (e.g., 77×768)
2. Cross-attention: The text embedding is injected into the U-Net via cross-attention layers
3. Conditioned denoising: The diffusion model uses the text embedding to guide image generation

In cross-attention (as in Transformer decoders): $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q_{image}K_{text}^T}{\sqrt{d}}\right)V_{text}$

Each image patch queries the text embedding to determine which words to attend to. This enables fine-grained text-image alignment. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: In CFG, guidance weight w=7 vs w=1.5. Compare the output characteristics.

w=7 (strong guidance):

• Very strong adherence to condition
• High saturation, contrast, and sharpness
• Less diversity (all samples look similar for same prompt)
• May produce artifacts or unnatural details
• Good for: "exactly what I described"

w=1.5 (weak guidance):

• More natural, diverse outputs
• Condition is followed but creatively
• Softer, more varied results
• Better image quality but possibly lower prompt adherence
• Good for: "creative interpretation of my prompt"

Typical values: Stable Diffusion uses w=7-9 by default. Imagen uses w=3-5. The optimal value depends on the model and desired trade-off between prompt alignment and image quality.

</details> * * * ## 🔗 Cross-References - **Next**: [Evaluation Metrics](/notes/02b-diploma-data-science-bscs2008-ml-practice-week08-08-evaluation-metrics) - **Previous**: [DDIM](/notes/04-degree-electives-bsda5002-genai-foundations-week05-05-ddim) - **Video**: BSDA5002 Week 7 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Autoregressive Models**](/notes/04-degree-electives-bsda5002-genai-foundations-week06-06-autoregressive)[Next**Evaluation Metrics**](/notes/04-degree-electives-bsda5002-genai-foundations-week08-08-evaluation-metrics)