Generative Models Overview: Taxonomy, Likelihood, and Fundamentals

🎯 Learning Objectives

• Understand what generative models are and how they differ from discriminative models
• Classify generative models into likelihood-based, implicit, and score-based categories
• Derive and interpret maximum likelihood estimation for generative modeling
• Explain the i.i.d. assumption and its implications for learning data distributions
• Compare the trade-offs between different generative modeling paradigms

📋 Prerequisites

• Probability distributions (BSMA1004): Random variables, PDF/PMF, conditional probability
• Basic calculus: Partial derivatives, chain rule
• Neural network basics: Forward pass, backpropagation

1. 📖 Core Content

1.1 Intuition: What is a Generative Model?

1. Recognize whether a new photo contains a cat (discrimination)
2. Draw a brand-new cat that looks realistic (generation) A generative model learns the second skill. It captures the underlying distribution of the training data so it can produce new samples that resemble the original data. Formally, if training data comes from an unknown true distribution $p_{\text{data}}(x)$, the generative model learns a distribution $p_\theta(x)$ that approximates $p_{\text{data}}(x)$. Why does this matter? Generative models power:

• Image generation: DALL·E, Midjourney, Stable Diffusion
• Text generation: GPT-4, Claude, Llama
• Drug discovery: Generating new molecular structures
• Data augmentation: Creating synthetic training examples
• Anomaly detection: Flagging inputs with low likelihood under the learned distribution

1.2 Discriminative vs Generative: The Fork in the Road

1.3 Taxonomy of Generative Models

Likelihood-based Models

• Autoregressive models (PixelCNN, GPT): Decompose $p(x) = \prod_t p(x_t | x_{<t})$. Easy to train, slow to sample.
• VAEs (Variational Autoencoders): Learn a latent variable model with an approximate posterior. Fast sampling, but blurry outputs.
• Normalizing Flows: Invertible transformations that map a simple base distribution to complex data. Exact likelihood, but computationally expensive.

Implicit Models

• GANs (Generative Adversarial Networks): Generator produces samples; discriminator judges realism. No explicit likelihood, sharp images, but training can be unstable.

Score-based / Diffusion Models

• Diffusion Models: Gradually add noise to data, then learn to reverse the process. State-of-the-art quality, slower sampling.
• Score Matching: Directly estimate $\nabla_x \log p(x)$.

1.4 Maximum Likelihood Estimation (MLE) for Generative Models

Intuition

Formal Definition

Why the Log?

• Converts product to sum (easier to differentiate)
• Prevents numerical underflow
• Monotonic transformation: maximizing log-likelihood is equivalent to maximizing likelihood

Worked Example 1: MLE for a Gaussian

Worked Example 2: MLE for Bernoulli

Worked Example 3: MLE for a Simple Generative Model

1.5 The i.i.d. Assumption

What It Says

1. Independent: The value of $x^{(i)}$ doesn't influence $x^{(j)}$ for $i \neq j$
2. Identically Distributed: Each $x^{(i)}$ comes from the same probability distribution $p_{\text{data}}$

Why We Need It

• With i.i.d.: $p(\mathcal{D}|\theta) = \prod_i p_\theta(x^{(i)})$ — simple product
• Without i.i.d.: $p(\mathcal{D}|\theta) = p_\theta(x^{(1)}) \cdot p_\theta(x^{(2)}|x^{(1)}) \cdot p_\theta(x^{(3)}|x^{(1)}, x^{(2)}) \cdot ...$ — much more complex

When i.i.d. Fails

• Time series: Stock prices today depend on yesterday's prices
• Spatial data: Adjacent pixels in an image are highly correlated
• Duplicate data: If the same data point appears multiple times, it's not independent

Edge Cases & Gotchas

• Small datasets: With few samples, the empirical distribution is a poor approximation of the true distribution
• Mode collapse: If the model memorizes training examples (especially common with small data), it fails to generalize
• Distribution shift: If test data comes from a different distribution than training data, the generative model will produce poor samples

1.6 Why This Matters

• Week 2 (GANs): Implicit models that avoid explicit likelihood
• Week 3 (VAEs): Likelihood-based models with tractable lower bounds
• Week 4 (Diffusion): Score-based models that learn gradients of log-density
• Week 7 (Conditional Generation): Controlling the output of any generative model The choice of which paradigm to use depends on your application: need fast sampling and don't care about exact likelihood? GANs. Need exact likelihood and have compute budget? Normalizing flows. Best quality? Diffusion models.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Confusing Generative and Discriminative

Pitfall 2: Believing Higher Likelihood Always Means Better Samples

Pitfall 3: Ignoring the i.i.d. Assumption When Processing Sequential Data

Pitfall 4: Maximizing Likelihood Without Regularization

4. 📝 Practice Questions

Q1: Is logistic regression a generative or discriminative model? Why?

Answer: Logistic regression is a discriminative model. It directly models $p(y|x)$ without modeling the distribution of x. It learns a decision boundary between classes, not the underlying data distribution. You cannot use logistic regression to generate new data points. Q2: Suppose you have a dataset of 5 numbers: [2, 3, 5, 7, 11]. Model it as $\mathcal{N}(\mu, \sigma^2=4)$. Find the MLE for $\mu$.

Step 1: Log-likelihood (dropping constant terms):

$\ell(\mu) = -\frac{1}{2\sigma^2} \sum_{i=1}^5 (x^{(i)}-\mu)^2 = -\frac{1}{8} \sum_{i=1}^5 (x^{(i)}-\mu)^2$

Step 2: Differentiate:

$\frac{d\ell}{d\mu} = \frac{1}{4} \sum_{i=1}^5 (x^{(i)}-\mu) = 0$

Step 3: Solve:

$(2+3+5+7+11) - 5\mu = 0$

$28 - 5\mu = 0$

$\hat{\mu} = \frac{28}{5} = 5.6$

The MLE is the sample mean: 5.6. Q3: Compare VAEs and GANs in terms of (a) likelihood computation, (b) sample quality, (c) training stability.

VAEs provide a tractable lower bound on likelihood but tend to produce blurry images due to the Gaussian decoder assumption and the KL regularizer. GANs can produce sharper images through adversarial training but don't provide likelihood estimates and are notoriously difficult to train due to the delicate balance between generator and discriminator. Q4: Why can't we directly compute the likelihood for a GAN?

GANs are implicit generative models. They define a sampling path: sample noise $z \sim p(z)$, then transform it through the generator $G_\theta(z)$ to produce a sample $x$. However, the generator defines a complex, high-dimensional transformation that makes it infeasible to compute $p_\theta(x)$ explicitly. The probability density would require integrating over all $z$ that map to $x$, which is intractable for neural network generators. Q5: A dataset has 1000 images, but 100 of them are exact duplicates. How does this violate the i.i.d. assumption?

Independence violation: Duplicates are not independent of each other — knowing one exists tells you the other exists. This artificially inflates the weight of those specific data points in the likelihood, causing the model to over-emphasize the duplicated content. The model effectively sees 900 unique images + 100 copies of the same image, making it believe that image is more important than it should be. Q6: Derive the MLE for the variance of a Gaussian (assuming known mean $\mu=0$) given data $\{x_1, ..., x_N\}$.

Step 1: Log-likelihood.

$\ell(\sigma^2) = \sum_{i=1}^N \left[-\frac{1}{2}\log(2\pi\sigma^2) - \frac{x_i^2}{2\sigma^2}\right]$

$= -\frac{N}{2}\log(2\pi) - \frac{N}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^N x_i^2$

Step 2: Differentiate w.r.t. $\sigma^2$:

$\frac{d\ell}{d(\sigma^2)} = -\frac{N}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_{i=1}^N x_i^2 = 0$

Step 3: Solve:

$-\frac{N}{2\sigma^2} + \frac{\sum x_i^2}{2\sigma^4} = 0$

Multiply by $2\sigma^4$: $-N\sigma^2 + \sum x_i^2 = 0$

$\hat{\sigma}^2 = \frac{1}{N}\sum_{i=1}^N x_i^2$

Note: This is a biased estimator (doesn't use Bessel's correction $N-1$). For small N, consider the unbiased version. Q7: Explain the "likelihood-free" training objective for GANs at a high level.

Instead of maximizing $p_\theta(x)$, GANs play a two-player game:

1. Discriminator $D$: Learns to distinguish real vs fake. Maximizes $\mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$.
2. Generator $G$: Learns to fool the discriminator. Minimizes $\mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$ (or equivalently maximizes $\mathbb{E}_{z}[\log D(G(z))]$).

At equilibrium, $G$ has learned $p_{\text{data}}$ without ever computing a likelihood. This is the key innovation: training a generative model through an adversarial discrimination game. Q8: What is the "curse of dimensionality" in the context of generative models?

In high-dimensional spaces (e.g., 256×256×3 images = 196,608 dimensions), the volume of space grows exponentially. This means:

1. Data sparsity: Fixed dataset covers an infinitesimally tiny fraction of the space
2. Density estimation failure: MLE pushes mass to observed points, leaving sparse regions with near-zero density
3. Sample quality degradation: The model may generate implausible samples in regions it never observed

Generative models combat this through:

• Manifold assumption: Real data lies on a low-dimensional manifold within the high-dimensional space
• Inductive biases: Convolutions (locality), recurrence (sequentiality), attention (long-range dependencies)
• Latent variable models: Learning compact representations (VAEs, GANs) Q9: For an autoregressive model modeling text as $p(x) = \prod_{t=1}^T p(x_t | x_{<t})$, is the i.i.d. assumption satisfied?

The data points within a single sequence (individual tokens $x_t$) are NOT independent — each token depends on all previous tokens. That's the point of the autoregressive decomposition.

However, different sequences in the dataset are assumed i.i.d. For example, each document in a text corpus is treated as independent and identically distributed.

The autoregressive factorization respects the within-sequence dependence while leveraging the between-sequence independence assumption for the likelihood computation. Q10: Give an example where a generative model is preferred over a discriminative model even though both can solve the task.

Anomaly Detection in Manufacturing: You have images of normal products (only one class). A generative model learns $p(x)$ for normal products. During testing, any product with low $p(x)$ is flagged as anomalous. A discriminative model would require examples of both normal AND anomalous products for training, which is often impossible because anomalies are rare and unpredictable.

Another example: Data Augmentation. When you have limited training data for a classification task, you can train a generative model (e.g., diffusion model) to create synthetic training examples, then use a discriminative model for the actual classification. The generative model enriches the training set. Q11: If a model's log-likelihood on test data is -1500 nats and another model's is -1400 nats, which is better and by how much?

The model with -1400 nats is better because it's a higher log-likelihood (less negative).

The difference of 100 nats means the better model assigns $e^{100} \approx 2.7 \times 10^{43}$ times higher probability to the test data. However, this comparison is only valid if both models are in the same family (both likelihood-based). You cannot directly compare a VAE's ELBO (-1400) with a normalizing flow's exact log-likelihood (-1500) because the VAE provides a lower bound, not the true likelihood. Q12: Design a simple experiment to test whether your generative model has memorized training data vs. learned to generalize.

Hold-out method: Split your dataset into training and held-out sets. Train on the training set. Now:

1. Compute likelihood: If test-set likelihood is close to training-set likelihood, the model generalizes well
2. Nearest neighbor test: For each generated sample, find its nearest neighbor in the training set. If most generated samples are nearly identical to training examples (e.g., pixel-wise MSE < threshold), the model is memorizing
3. Interpolation test: Generate samples that should interpolate between training examples. If they look like unrealistic blends, the model has memorized specific points rather than learned the manifold

Ideally, a good generative model produces novel samples that resemble training data without being copies.

5. 🔗 Cross-References

• Next: GANs (Week 2) — Implicit generative models
• Related: VAEs (Week 3) — Variational approach to likelihood-based modeling
• Related: Diffusion Models (Week 4) — Score-based generative models
• External: IITM BS Degree Statistics 2 course notes on MLE Join Discord NextGANs