Evaluation Metrics for Generative Models: FID, IS, Precision-Recall, and Diversity

🎯 Learning Objectives

• Understand why evaluating generative models is fundamentally different from discriminative models
• Compute and interpret Fréchet Inception Distance (FID)
• Analyze the strengths and weaknesses of the Inception Score (IS)
• Evaluate generative models along precision and recall dimensions separately
• Implement diversity metrics and detect mode collapse
• Choose appropriate metrics for different generative model types (GANs, VAEs, Diffusion)

📋 Prerequisites

• Generative Models Overview (Week 1): Understanding of what generative models produce
• Basic statistics: Mean, variance, multivariate Gaussians, KL divergence
• Image classification: Understanding of feature representations (e.g., from CNNs)

1. 📖 Core Content

1.1 Intuition: Why Can't We Just Use Accuracy?

1. Fidelity: Do generated samples look realistic? (quality)
2. Diversity: Do generated samples cover all modes of the data? (coverage)
3. Novelty: Are generated samples new, or are they memorized training examples?
4. Likelihood: How high is the probability assigned to held-out data? No single metric captures all four. Worse, some metrics can be gamed — a model that memorizes training data gets perfect fidelity but zero novelty.

1.2 Fréchet Inception Distance (FID)

1.2.1 Intuition

1.2.2 Formal Definition

• $\mu_r, \Sigma_r$: Mean and covariance of real image features
• $\mu_g, \Sigma_g$: Mean and covariance of generated image features
• $\text{Tr}$: Trace of the matrix
• $\sqrt{\Sigma_r \Sigma_g}$: Matrix square root of $\Sigma_r \Sigma_g$ Lower FID is better (smaller distance between distributions).

1.2.3 Worked Example 1: Computing FID for Simple 2D Features

1.2.4 Practical Guidelines for FID

• FID requires many samples (typically 50K) for stable estimates
• FID depends on the training set of the Inception network — it's biased toward ImageNet-like images
• FID can be gamed by generating images with similar Inception features but different visual appearance
• Small sample sizes produce unreliable FID estimates

1.3 Inception Score (IS)

1.3.1 Intuition

1. Contain clear objects: The Inception network confidently classifies them (low entropy of $p(y|x)$)
2. Cover many classes: The generated set has diverse classes (high entropy of $p(y)$) IS combines these two desiderata.

1.3.2 Formal Definition

• $p(y|x)$ is the Inception network's label distribution for generated image $x$
• $p(y) = \mathbb{E}_{x \sim p_g}[p(y|x)]$ is the marginal label distribution
• $D_{KL}$ is the KL divergence Equivalently:

1.3.3 Worked Example 2: Computing IS for Two Models

1.3.4 Limitations of IS

1. No real-world reference: IS only looks at generated samples, comparing them to ImageNet classes. A model generating only one class per image but all classes equally gets a high IS regardless of whether images look real.
2. Relies on Inception: Biased toward ImageNet-like images; doesn't work well for medical images, satellite imagery, etc.
3. Insensitive to mode dropping: A model that generates only 10 out of 1000 ImageNet classes can still get a reasonable IS if those 10 classes are diverse.
4. Doesn't detect overfitting: Memorized training images can get high IS.

1.4 Precision and Recall for Generative Models

1.4.1 Intuition

• Precision: What fraction of generated images look realistic?
• Recall: What fraction of real-data modes are covered by the generated distribution? A model that generates perfect cats but no dogs has high precision (cat images look real) but low recall (missing dog mode).

1.4.2 The Improvement over FID

1.4.3 How It Works (Kynkäänniemi Method)

1.4.4 Worked Example 3: Precision-Recall Analysis

• Precision: 1000/1000 = 1.0 (all generated look real)
• Recall: The cat and dog modes are covered; the bird mode is not. So recall = 2/3 ≈ 0.67. Model Y: Generates 1000 images across all 3 modes, but 200 look unrealistic.
• Precision: 800/1000 = 0.8 (200 unrealistic)
• Recall: 1000/1000 = 1.0 (all real modes covered) Model Z: Generates only 1 image (a perfect cat), repeated 1000 times.
• Precision: 1000/1000 = 1.0 (all look realistic)
• Recall: 1/3 ≈ 0.33 (dog and bird modes missing) The precision-recall decomposition reveals these trade-offs that FID alone would mask.

1.5 Diversity Metrics

1.5.1 Perceptual Path Length (PPL)

1.5.2 Intra-Class Diversity

1.6 Likelihood-Based Evaluation

• PixelCNN++: ~2.92 bits/dim on CIFAR-10
• Glow (normalizing flow): ~3.35 bits/dim on CIFAR-10
• VAE: ~3.70 bits/dim on CIFAR-10 Important: Don't compare bits/dim across datasets. Also, bits/dim doesn't always correlate with perceptual quality — some models with good bits/dim produce poor-looking samples.

1.7 Metric Comparison Summary

1.8 Best Practices

1. Always report multiple metrics: FID + Precision-Recall + IS covers quality, diversity, and real-world correspondence.
2. Use sufficient samples: FID with < 10K samples is unreliable. 50K is standard.
3. Control for model seed: Different random seeds can produce different FIDs. Report mean + std over multiple runs.
4. Don't compare across papers directly: Implementation details (number of samples, preprocessing, feature extraction layer) vary. Always compare within your own setup.
5. Check for memorization: Use nearest-neighbor analysis to ensure generated images aren't just training set copies.
6. Use dataset-specific metrics: For non-ImageNet datasets, use a feature extractor fine-tuned on that domain.

1.9 Why This Matters

• Comparing GAN variants: FID + Precision-Recall (best overall)
• Debugging training: Precision-Recall helps identify whether quality or diversity is the problem
• Production deployment: FID + human evaluation (metrics alone aren't enough)
• Academic research: FID + IS + bits/dim (if applicable) + precision-recall

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Using FID with Too Few Samples

Pitfall 2: Using Inception Score for Non-ImageNet Domains

Pitfall 3: Relying Solely on FID for Model Selection

Pitfall 4: Comparing FIDs Across Different Preprocessing Pipelines

• Which layer of Inception is used (pool3 vs. logits)
• Image preprocessing (resizing method, normalization)
• Number of samples
• Which real-data reference set is used Correct approach: Reproduce the exact evaluation setup from the paper you're comparing against. When reporting your own, specify all implementation details.

Pitfall 5: Ignoring Memorization When Reporting Likelihood

4. 📝 Practice Questions

Q1: A GAN generates 50,000 images of faces. Compute FID: $\mu_g = [0.5, -0.2]$, $\Sigma_g = [1.1, 0.1], [0.1, 0.9](/courses/bsda5002/notes/1.1$. Real data: $\mu_r = [0.3, 0.1]$, $\Sigma_r = [1.0, 0.0], [0.0, 1.0](/courses/bsda5002/notes/1.0$. Calculate FID.

Step 1: $\|\mu_r - \mu_g\|^2 = (0.3-0.5)^2 + (0.1+0.2)^2 = 0.04 + 0.09 = 0.13$

Step 2: $\text{Tr}(\Sigma_r + \Sigma_g) = \text{Tr}([2.1, 0.1], [0.1, 1.9](/courses/bsda5002/notes/2.1$

Step 3: $\Sigma_r \Sigma_g = [1.0, 0.0], [0.0, 1.0](/courses/bsda5002/notes/1.0$

Step 4: Compute eigenvalues: $(1.1-\lambda)(0.9-\lambda) - 0.01 = \lambda^2 - 2\lambda + 0.99 - 0.01 = \lambda^2 - 2\lambda + 0.98 = 0$

$\lambda = \frac{2 \pm \sqrt{4 - 3.92}}{2} = \frac{2 \pm \sqrt{0.08}}{2} = \frac{2 \pm 0.283}{2}$

$\lambda_1 = 1.1415$, $\lambda_2 = 0.8585$

Step 5: $\text{Tr}(\sqrt{\Sigma_r \Sigma_g}) = \sqrt{1.1415} + \sqrt{0.8585} = 1.068 + 0.927 = 1.995$

Step 6: FID = $0.13 + 4.0 - 2(1.995) = 0.13 + 4.0 - 3.99 = 0.14$

Very low FID (0.14) — the generated distribution closely matches the real one. Q2: A model generates 5 dog breeds out of 100 ImageNet classes equally, with avg confidence of 0.95 per generated image. Another generates all 100 classes equally but with avg confidence of only 0.3. Which has higher IS?

Model A (5 breeds, high confidence):

• $H(p(y))$: Uniform over 5 classes → $\log(5) \approx 1.61$ nats
• $\mathbb{E}[H(p(y|x))]$: Per-image entropy ≈ $-(0.95\log(0.95) + 0.05\log(0.05/99)...)$ ≈ very low ≈ 0.3 nats
• KL = $1.61 - 0.3 = 1.31$
• IS = $\exp(1.31) \approx 3.7$

Model B (100 classes, low confidence):

• $H(p(y))$: Uniform over 100 classes → $\log(100) \approx 4.61$ nats
• $\mathbb{E}[H(p(y|x))]$: Per-image entropy ≈ $-(0.3\log(0.3) + 0.7\log(0.7/99)...)$ ≈ 2.8 nats
• KL = $4.61 - 2.8 = 1.81$
• IS = $\exp(1.81) \approx 6.1$

Model B has higher IS despite lower per-image confidence, because it covers more diverse classes. The IS favors diversity very strongly — this is both a strength and a weakness. Q3: A model produces perfect samples of 8 out of 10 real data modes. The other 2 modes are completely missing. Do precision and recall capture this?

• Precision: All generated samples look realistic (they only come from the 8 modes the model learned). Precision ≈ 1.0.
• Recall: Only 8 out of 10 real modes are covered. Recall ≈ 0.8.

FID would be moderately good (features match well on 8/10 modes but the missing 2 contribute to mean/covariance difference). The precision-recall decomposition reveals that the model has a coverage problem (low recall) that FID alone doesn't clearly indicate.

If the missing 2 modes are small (few training examples), FID might barely penalize the model, while recall catches it by checking whether real-data points lie in the generated-data manifold. Q4: Why does FID use the Inception network's pooling layer (2048-d features) instead of the logits (1000-d class probabilities)?

The pooling layer preserves spatial and structural information that the logits discard. Logits collapse the representation to class probabilities, losing information such as:

• Object pose, orientation, and position
• Background characteristics
• Fine-grained textures that aren't class-discriminative

Using the pooling layer's 2048-dimensional feature vector captures more of these visual characteristics, making FID more sensitive to visual quality differences that don't affect classifiability.

For example, two cat images (both classified as "tabby cat" with 0.95 confidence) could have very different visual quality — blurry vs sharp — which the pooling layer features reflect but logits don't. Q5: You train a VAE and get bits/dim = 3.5 on CIFAR-10 test set. What does this mean in terms of compression?

Bits/dim measures how many bits on average are needed to encode each dimension (here, each color channel of each pixel) under the model. A bits/dim of 3.5 means:

• Per pixel (3 color channels): $3 \times 3.5 = 10.5$ bits
• Per 32×32 image: $32 \times 32 \times 10.5 = 10752$ bits ≈ 1.34 KB

For comparison:

• Uncompressed PNG: ~3-16 bits/pixel (~9-48 KB for CIFAR-10)
• State-of-the-art compression (BPG): ~0.5-2 bits/pixel

The model compresses CIFAR-10 images to about 1.34 KB each on average under the lossless coding scheme derived from the model. A lower bits/dim means the model better captures the data distribution, achieving higher compression. Q6: Explain why the Inception Score doesn't require real data while FID does. What are the consequences?

IS only uses generated data: It computes $p(y|x)$ (Inception distribution for each generated image) and $p(y)$ (marginal over all generated images). No real data is needed.

FID requires both real and generated data: It compares feature distributions between the two.

Consequences:

• IS can be computed without access to the training set (useful for evaluating models trained on private data)
• IS doesn't penalize "unrealistic but classifiable" images — a generated image could look nothing like a real image but still get high IS if the Inception network confidently classifies it
• IS is blind to certain failure modes: a model generating only 5 of 1000 classes can get good IS
• FID is more robust because it directly compares to real data, but requires access to real data statistics

The best practice is to use both: IS as a measure of intra-sample quality + diversity, FID as a measure of distributional match to real data. Q7: You're developing a generative model for medical X-ray images. What evaluation metrics would you use and why?

Not recommended for this domain:

• Inception Score: The Inception network is trained on ImageNet (natural images). X-ray images don't match any ImageNet class, so IS would be meaningless.
• FID with standard Inception: Same issue — Inception features don't capture medically relevant features.

Recommended:

1. Domain-specific FID: Fine-tune a feature extractor (e.g., ResNet-50) on a large X-ray dataset (e.g., CheXpert). Use its penultimate layer features for FID computation.
2. Precision-Recall: Using the domain-specific features — especially important for medical imaging where some pathologies (modes) are rare.
3. Freight check by domain experts: Metrics for medical images must correlate with clinical utility. Have radiologists evaluate generated images for diagnostic consistency.
4. Diagnostic utility metric: Train a downstream classifier (e.g., pneumonia detector) on real + generated data and measure if generated data improves classifier performance.

Critical consideration: In medical imaging, generating a plausible but incorrect X-ray could be dangerous. FID and IS don't catch radiologically meaningful errors. Always include human expert evaluation. Q8: How can you detect whether a generative model is memorizing training data rather than generalizing?

A healthy generative model should have:

• Generated test nearest neighbors that are visually similar but not identical
• Training and test bits/dim within a small gap (e.g., < 10% difference)
• Generated images that humans can't systematically distinguish from unseen real data Q9: A diffusion model achieves FID=8.0 while a GAN achieves FID=6.5 on the same dataset. Can we conclude the GAN is better?

Not definitively. FID is one metric among many, and several factors complicate direct comparison:

1. FID variance: FID has stochasticity from sampling. With 50K samples, a difference of 1.5 FID points may or may not be statistically significant. Bootstrap confidence intervals should be computed.
2. Precision-Recall tradeoff: The GAN might have better aggregate FID but lower recall (missing some modes), while the diffusion model might have better coverage.
3. Other metrics: Check IS, precision-recall, human evaluation. A diffusion model with slightly worse FID might produce more clinically acceptable samples.
4. Application context: For creative tools, you might prefer the GAN's sharper images. For scientific applications requiring diversity, you might prefer the diffusion model.
5. Computational cost: The diffusion model might require 1000 steps vs. the GAN's 1 step. In a resource-constrained setting, the GAN might be practically superior despite a small FID disadvantage.

Always use multiple metrics and inspect samples visually before drawing conclusions. Q10: Explain the concept of "manifold" in precision-recall evaluation for generative models.

In precision-recall for generative models, the manifold is the set of feature vectors that represent the "typical" data points. It's estimated from samples:

Real-data manifold: The region in feature space where real data points fall. Computed by:

1. For each real data point, find its k nearest neighbors among real data
2. The manifold is the union of hyperspheres with radius = distance to the k-th neighbor

Generated-data manifold: Same procedure but for generated data points.

Precision = fraction of generated points that fall inside the real-data manifold.

• High precision means generated samples have features similar to real data
• Low precision means generated samples have "unrealistic" features

Recall = fraction of real data points that fall inside the generated-data manifold.

• High recall means the generator covers all regions where real data exists
• Low recall means some real data regions are not covered (mode dropping)

This approach implicitly handles multi-modal distributions because each data point has its own nearest-neighbor distance, which adapts to local data density. Q11: Why does FID sometimes improve when you train a model longer, even though sample quality appears to degrade visually?

This is known as the FID-quality mismatch and can happen for several reasons:

1. Feature space collapse: The model learns to generate features that match the aggregate statistics (mean + covariance) of real data in Inception space without generating visually realistic samples. Think of it as "cheating" on the FID exam by matching features rather than images.
2. Inception bias: The Inception network focuses on class-discriminative features. A model could learn to produce good class-specific features while generating implausible textures or backgrounds.
3. Overfitting to FID: If you select checkpoints based on FID, you're optimizing toward FID. The FID landscape might not perfectly correlate with human perceptual quality.

To detect this: always inspect generated samples visually alongside FID. If FID improves but visual quality degrades, there's a problem with your evaluation setup. In such cases, use precision-recall to check if the FID improvement comes from better recall (more coverage) but worse precision (lower individual quality). Q12: You have a model with precision=0.85 and recall=0.60. Propose two strategies to improve each.

To improve precision (0.85 → higher):

1. Reduce truncation: If using a truncation trick (common in StyleGAN), increase the truncation parameter ψ (e.g., from 0.7 to 0.5). This reduces variation but improves per-sample quality by sampling closer to the mean of the latent distribution.
2. Increase discriminator capacity (if GAN): A stronger discriminator forces the generator to produce more realistic samples, improving precision at the potential cost of recall.
3. Tune training hyperparameters: Lower learning rate, increase gradient penalty coefficient (WGAN-GP), or adjust regularization to reduce training instability that causes bad samples.

To improve recall (0.60 → higher):

1. Increase latent dimension: More noise dimensions can help the generator cover more modes.
2. Reduce truncation: If using truncation, decrease ψ (e.g., from 0.7 to 1.0). This allows more variation but may introduce lower-quality samples.
3. Data augmentation: Use more aggressive augmentation to prevent the generator from focusing on a subset of modes.
4. Re-balance GAN training: If the discriminator is too strong, it may prevent the generator from exploring new modes. Weaken the discriminator (dropout, fewer parameters) to give the generator more freedom.
5. Use a mixture model: Instead of a single generator, use multiple generators specializing in different modes.

Typically, precision and recall have an inverse relationship — improving one often harms the other. The goal is to find the right balance for your application.

5. 🔗 Cross-References

• Previous: Autoregressive Models (Week 6) — Autoregressive density estimation
• Next: Information Theory (Week 9) — Entropy, KL divergence foundations
• Related: Conditional Generation (Week 7) — Conditional FID and evaluation
• Related: Math Foundations (Week 9) — Multivariate Gaussians and eigendecomposition for FID
• External: Heusel et al., "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium" (NIPS 2017)
• External: Kynkäänniemi et al., "Improved Precision and Recall Metric for Assessing Generative Models" (NeurIPS 2019) Join Discord PreviousConditional GenerationNextInformation Theory