Information Theory: Entropy, KL Divergence, Mutual Information, and Cross-Entropy

🎯 Learning Objectives

• Define entropy and interpret it as a measure of uncertainty
• Derive and apply KL divergence for comparing probability distributions
• Compute mutual information between random variables
• Understand the relationship between cross-entropy, KL divergence, and log-likelihood
• Apply information-theoretic concepts to generative model analysis
• Explain how information theory connects to ELBO, GAN training, and diffusion models

📋 Prerequisites

• Probability basics: Random variables, PDF/PMF, expectation
• Generative Models Overview (Week 1): Likelihood, model distributions
• Basic calculus: Logarithms, differentiation

1. 📖 Core Content

1.1 Intuition: What is Information?

• "The sun rose this morning" — highly probable, low information
• "An earthquake hit Mumbai this morning" — low probability, high information An event with certainty carries zero information:
• "A fair coin will land either heads or tails" — 100% certain, 0 bits of information In generative modeling, information theory quantifies:
• How surprised is the model by new data? (cross-entropy)
• How different is the learned distribution from the real one? (KL divergence)
• How much does the input inform the output? (mutual information)

1.2 Self-Information

• Base 2: bits (most common in information theory)
• Base $e$: nats (most common in ML, because gradients are easier)
• Base 10: dits (rare)

Worked Example 1: Computing Self-Information

1.3 Entropy: Expected Information

1.3.1 Interpretation

• High entropy → Many possible outcomes, roughly equally likely
• Low entropy → Outcome is fairly predictable

Worked Example 2: Comparing Entropies

1.3.2 Maximum Entropy Principle

• Gaussian noise models "maximum ignorance" — it's the least informative noise given first and second moments
• Diffusion models add Gaussian noise because it's the most entropic (most destructive) noise

1.4 KL Divergence: Measuring Distribution Difference

1.4.1 Key Properties

1. Non-negativity: $D_{KL}(p \| q) \geq 0$ (Gibbs' inequality). Equality iff $p = q$ almost everywhere.
2. Not symmetric: $D_{KL}(p \| q) \neq D_{KL}(q \| p)$ in general.
3. Not a metric: Doesn't satisfy symmetry or triangle inequality.

1.4.2 Asymmetry in Practice

• Forward KL $D_{KL}(p \| q)$: Average over $p$ of $\log(p/q)$. This forces $q > 0$ wherever $p > 0$. Result: $q$ tries to cover all modes of $p$ (mean-seeking).
• Reverse KL $D_{KL}(q \| p)$: Average over $q$ of $\log(q/p)$. This forces $q$ to place probability mass only where $p > 0$. Result: $q$ tries to fit a single mode of $p$ (mode-seeking).

Worked Example 3: KL Divergence Between Two Gaussians

Worked Example 4: Forward vs Reverse KL for a Bimodal Distribution

• We average over $p$, so $q$ must have support everywhere $p$ has mass
• $q$ will try to cover both modes
• Optimal $q$: $\mu \approx 0$, $\sigma \approx 3$ (high variance to cover both peaks, but puts probability mass between the modes where $p$ has none) Reverse KL $D_{KL}(q \| p)$:
• We average over $q$, so $q$ tries to avoid regions where $p$ is very small
• Since $p$ has a valley at 0, $q$ will choose to model a single mode
• Optimal $q$: either $\mu \approx -2$, $\sigma \approx 1$ (left mode) or $\mu \approx 2$, $\sigma \approx 1$ (right mode) This explains why:
• VAEs minimize reverse KL ($q(z|x)$ is optimized to avoid areas where $p(z)$ has low density)
• Maximum likelihood estimation minimizes forward KL ($p_\theta(x)$ must cover all modes of $p_{\text{data}}$)

1.5 Cross-Entropy

1.5.1 Cross-Entropy Loss in ML

1.6 Mutual Information

1.6.1 Interpretation

• I(X;Y) = 0: X and Y are independent (knowing X tells nothing about Y)
• I(X;Y) = H(X): Y perfectly predicts X (and vice versa)
• Higher MI → Stronger dependence between variables

Worked Example 5: Mutual Information for a Noisy Channel

• Input: $X \in \{0, 1\}$, $P(X=0) = P(X=1) = 0.5$
• Channel: transmits correctly with probability 0.9, flips with probability 0.1 Step 1: Compute $H(X)$:

1.7 Applications in Generative Models

1.7.1 ELBO Decomposition

1.7.2 InfoGAN: Maximizing Mutual Information

1.7.3 Diffusion Model ELBO

1.8 Edge Cases & Gotchas

• KL divergence can be infinite: If $q(x) = 0$ but $p(x) > 0$, then $D_{KL}(p \| q) = \infty$. This is why VAEs use reverse KL (where $q$ is the approximate posterior and we ensure it has broad support).
• Differential entropy can be negative: Unlike discrete entropy, continuous entropy can be negative (e.g., a very narrow Gaussian has low entropy).
• Mutual information is bounded: $0 \leq I(X;Y) \leq \min(H(X), H(Y))$.
• Data processing inequality: Post-processing can't increase information. If $X \to Y \to Z$, then $I(X;Z) \leq I(X;Y)$.

1.9 Why This Matters

1. Diagnose training failures (e.g., KL vanishing in VAEs)
2. Design new model variants (e.g., β-VAE that weights KL)
3. Choose appropriate evaluation metrics

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Forgetting KL Divergence is Asymmetric

Pitfall 2: Confusing Cross-Entropy with KL Divergence

Pitfall 3: Using Bits Instead of Nats

Pitfall 4: Ignoring KL Vanishing in VAEs

Pitfall 5: Misinterpreting Mutual Information for High-Dimensional Data

4. 📝 Practice Questions

Q1: Compute the entropy of a 6-sided fair die in bits.

$H = -\sum_{i=1}^6 \frac{1}{6}\log_2(\frac{1}{6}) = \log_2(6) \approx 2.585 \text{ bits}$

This means on average, it takes about 2.585 bits to encode the outcome of a fair 6-sided die. Q2: Two coins: Coin A is fair ($p=0.5$), Coin B is biased ($p=0.99$). Which has higher entropy and why?

Coin A: $H = -0.5\log_2(0.5) - 0.5\log_2(0.5) = 1.0 \text{ bit}$

Coin B: $H = -0.99\log_2(0.99) - 0.01\log_2(0.01) \approx 0.081 \text{ bits}$

Coin A has higher entropy because its outcome is more uncertain. Coin B is highly predictable (almost always heads), so its entropy is near zero. Q3: Compute $D_{KL}(p \| q)$ where $p$ is Bernoulli(0.5) and $q$ is Bernoulli(0.8).

$D_{KL}(p \| q) = \sum_{x \in \{0,1\}} p(x) \log\frac{p(x)}{q(x)}$

$= 0.5 \log\frac{0.5}{0.8} + 0.5 \log\frac{0.5}{0.2}$

$= 0.5 \log(0.625) + 0.5 \log(2.5)$

$= 0.5(-0.470) + 0.5(0.916)$

$= -0.235 + 0.458 = 0.223 \text{ nats}$

In bits: $0.223 \times \log_2(e) \approx 0.223 \times 1.44 \approx 0.321 \text{ bits}$ Q4: In the VAE loss function, why does the KL term use $D_{KL}(q_\phi(z|x) \| p(z))$ instead of $D_{KL}(p(z) \| q_\phi(z|x))$?

The ELBO derivation starts from:

$\log p(x) = D_{KL}(q_\phi(z|x) \| p_\theta(z|x)) + \mathcal{L}$

where $\mathcal{L}$ is the ELBO. We want to minimize the KL between $q_\phi(z|x)$ and the true posterior $p_\theta(z|x)$, but we can't compute $p_\theta(z|x)$ directly. So we rearrange:

$\mathcal{L} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) \| p(z))$

The KL term is $D_{KL}(q_\phi(z|x) \| p(z))$ — we minimize this to make the approximate posterior close to the prior. This is reverse KL because we average over the variational distribution $q_\phi$.

Using reverse KL means $q_\phi(z|x)$ tends to be mode-seeking — it will try to find a single good latent code for each $x$ rather than covering all possible latent explanations. This is actually desirable: for a given input, we want a compact, informative latent representation.

If we used forward KL $D_{KL}(p(z) \| q_\phi(z|x))$ instead, the KL would be $-\infty$ wherever $q_\phi$ assigns zero probability where $p(z) > 0$, which would force $q_\phi$ to have very broad support for every input — undesirable. Q5: In InfoGAN, the mutual information $I(c; G(z,c))$ is maximized. Show how the variational lower bound $I(c; x) \geq \mathbb{E}_{x \sim G(z,c)}[\log q(c|x)] + H(c)$ is derived.

Starting from the definition:

$I(c; x) = H(c) - H(c|x) = H(c) + \mathbb{E}_{x \sim G(z,c)}[\mathbb{E}_{c' \sim p(c|x)}[\log p(c'|x)]]$

We introduce a variational approximation $q(c|x)$ to the true posterior $p(c|x)$:

$I(c; x) = H(c) + \mathbb{E}_{x}[\mathbb{E}_{c' \sim p(c|x)}[\log p(c'|x)]]$

$= H(c) + \mathbb{E}_{x}[\mathbb{E}_{c' \sim p(c|x)}[\log q(c'|x)] + D_{KL}(p(c|x) \| q(c|x))]$

Since $D_{KL} \geq 0$:

$I(c; x) \geq H(c) + \mathbb{E}_{x \sim G(z,c)}[\log q(c|x)]$

This gives a tractable lower bound on mutual information that we can optimize. $q(c|x)$ is an auxiliary network that tries to predict the latent code $c$ from the generated image $x$. Q6: A Gaussian distribution $\mathcal{N}(0, 0.1)$ has differential entropy. Compute it.

Differential entropy of a univariate Gaussian:

$H(\mathcal{N}(\mu, \sigma^2)) = \frac{1}{2}\ln(2\pi e \sigma^2)$

For $\sigma^2 = 0.1$:

$H = \frac{1}{2}\ln(2\pi e \times 0.1) = \frac{1}{2}\ln(0.2\pi e) = \frac{1}{2}\ln(1.708) \approx \frac{1}{2} \times 0.536 = 0.268 \text{ nats}$

This is positive (low entropy for a narrow distribution). For a very narrow Gaussian ($\sigma \to 0$), differential entropy is $-\infty$. Q7: Explain why bits/dim is equivalent to cross-entropy per dimension.

Bits/dim measures the average number of bits needed to encode each dimension of the data under the model. For an image of $D = H \times W \times C$ dimensions:

$\text{bits/dim} = -\frac{\log_2 p_\theta(x)}{D} = \frac{1}{D} \cdot \underbrace{(-\log_2 p_\theta(x))}_{\text{cross-entropy in bits}}$

This is the cross-entropy between the true data distribution (which places a delta on $x$) and the model, normalized by dimensionality.

Bits/dim allows comparison across models trained on the same dataset, regardless of image resolution. A bits/dim of 3.0 means the model compresses each pixel channel (or each data dimension) to 3 bits on average. Q8: $X$ is uniformly distributed over $\{1, 2, 3, 4\}$. $Y = X^2 \mod 4$. Compute $I(X; Y)$.

Step 1: $H(X) = \log_2(4) = 2 \text{ bits}$

Step 2: $Y = X^2 \mod 4$:

• $X=1 \to Y=1$
• $X=2 \to Y=0$
• $X=3 \to Y=1$
• $X=4 \to Y=0$

So $P(Y=0) = 0.5$, $P(Y=1) = 0.5$

Step 3: $H(Y) = 1 \text{ bit}$

Step 4: $P(X|Y=0)$: $X \in \{2, 4\}$ equally likely → $H(X|Y=0) = 1 \text{ bit}$ $P(X|Y=1)$: $X \in \{1, 3\}$ equally likely → $H(X|Y=1) = 1 \text{ bit}$

$H(X|Y) = 0.5 \times 1 + 0.5 \times 1 = 1 \text{ bit}$

Step 5: $I(X; Y) = H(X) - H(X|Y) = 2 - 1 = 1 \text{ bit}$

Knowing $Y$ reduces the uncertainty about $X$ from 2 bits to 1 bit — you still don't know whether $X$ is the smaller or larger value in the pair. Q9: Why does adding noise increase entropy? Use this to explain the forward process in diffusion models.

Adding noise increases entropy because it makes the distribution more uncertain. In the forward diffusion process:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t)I)$

As $t$ increases (more noise added):

• $\bar{\alpha}_t \to 0$
• Mean goes to 0
• Variance goes to $I$

The entropy of $q(x_t | x_0)$ increases from near-zero (at $t=0$, $x_t$ is nearly deterministic given $x_0$) to:

$H(q(x_T)) \approx \frac{D}{2}\ln(2\pi e)$

which is the maximum entropy for a $D$-dimensional distribution with fixed variance — matching a standard Gaussian $\mathcal{N}(0, I)$.

This increase in entropy is why diffusion models can destroy any data distribution: by gradually adding noise, we wash away the structure until only Gaussian noise remains. Q10: The $\beta$-VAE modifies the VAE loss to $\mathcal{L} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - \beta \cdot D_{KL}(q_\phi(z|x) \| p(z))$. Explain why $\beta > 1$ encourages more disentangled representations.

With $\beta > 1$, the KL term is weighted more heavily, which pushes each latent dimension's distribution $q_\phi(z_i|x)$ to be closer to the prior $p(z_i) = \mathcal{N}(0, 1)$.

For disentanglement, this means:

1. Factorized prior: The prior $p(z) = \prod_i p(z_i)$ is factorized (independent dimensions). By pushing $q_\phi(z|x)$ toward $p(z)$, we encourage the posterior to also be factorized.
2. Information bottleneck: The higher KL penalty limits the total information that $z$ can carry about $x$. The model must allocate this limited information efficiently, learning to encode only the most salient, independent factors of variation in separate dimensions.
3. Selective pressure: Dimensions that aren't informative are "turned off" (their KL matches the prior exactly), leaving only the truly informative dimensions active.

However, $\beta > 1$ also reduces reconstruction quality (the rate-distortion tradeoff). Higher $\beta$ reduces "rate" (KL) but increases "distortion" (reconstruction error). Finding the right $\beta$ balances disentanglement with reconstruction fidelity. Q11: Show that maximizing likelihood is equivalent to minimizing the forward KL divergence between the empirical data distribution and the model distribution.

The empirical data distribution is:

$\hat{p}_{\text{data}}(x) = \frac{1}{N} \sum_{i=1}^N \delta(x - x^{(i)})$

where $\delta$ is the Dirac delta.

Maximum likelihood estimation maximizes:

$\frac{1}{N} \sum_{i=1}^N \log p_\theta(x^{(i)}) = \mathbb{E}_{x \sim \hat{p}_{\text{data}}}[\log p_\theta(x)]$

The forward KL divergence between $\hat{p}_{\text{data}}$ and $p_\theta$:

$D_{KL}(\hat{p}_{\text{data}} \| p_\theta) = \mathbb{E}_{x \sim \hat{p}_{\text{data}}}[\log \hat{p}_{\text{data}}(x) - \log p_\theta(x)]$

$= \underbrace{\mathbb{E}_{x \sim \hat{p}_{\text{data}}}[\log \hat{p}_{\text{data}}(x)]}_{\text{constant (entropy of empirical dist)}} - \underbrace{\mathbb{E}_{x \sim \hat{p}_{\text{data}}}[\log p_\theta(x)]}_{\text{log-likelihood}}$

Since the first term is constant with respect to $\theta$, minimizing $D_{KL}$ is equivalent to maximizing the log-likelihood.

This is forward KL because we average over $\hat{p}_{\text{data}}$ — the model must assign high probability to every data point, which forces it to cover all modes of the data distribution. Q12: For the KL divergence $D_{KL}(p \| q)$ where $p$ is a mixture of two Gaussians and $q$ is a single Gaussian, sketch the result under forward KL vs reverse KL optimization. Which one would produce a better generative model for image data?

When $p$ is bimodal (two modes) and $q$ is a single Gaussian:

Forward KL ($D_{KL}(p \| q)$ minimized over $q$):

• $q$ spreads out to cover both modes
• Result: high-variance Gaussian centered between the modes
• $q$ has mass in the middle where $p$ has none
• This is "mean-seeking" — it captures the overall spread but not the structure

Reverse KL ($D_{KL}(q \| p)$ minimized over $q$):

• $q$ picks one mode and fits it tightly
• Result: a narrow Gaussian centered on one mode
• The other mode is completely ignored
• This is "mode-seeking" or "mode-collapse"

For image data, forward KL is generally preferred because:

1. Real images have many modes (cats, dogs, cars, etc.)
2. Forward KL forces the model to cover all modes (better recall)
3. Spread-out blurry images are better than missing entire categories

However, forward KL tends to produce blurry images (the model averages over plausible outputs). This is why VAEs (trained with forward KL through MLE) produce blurry images, while GANs (which use a different loss) can produce sharp images but may suffer from mode collapse (a reverse-KL-like failure mode).

This also explains why diffusion models (trained with forward KL via NLL) can be slightly blurry with few steps but sharp with enough steps — the iterative refinement allows the model to escape the blurry mean prediction.

5. 🔗 Cross-References

• Previous: Evaluation Metrics (Week 8) — Cross-entropy as evaluation metric
• Next: Math Foundations (Week 9) — SVD, eigendecomposition for information theory
• Related: VAEs (Week 3) — ELBO, KL divergence in practice
• Related: Diffusion Models (Week 4) — KL between Gaussians in diffusion loss
• External: Cover & Thomas, "Elements of Information Theory" — Comprehensive reference Join Discord PreviousEvaluation MetricsNextMath Foundations