Math Foundations: SVD, Eigendecomposition, Matrix Calculus, and ELBO Derivation

🎯 Learning Objectives

• Compute and interpret singular value decomposition (SVD) and eigendecomposition
• Understand matrix calculus for gradient-based optimization
• Derive the ELBO from first principles with full mathematical detail
• Apply optimization concepts to generative model training
• Connect linear algebra to key generative model operations

📋 Prerequisites

• Linear algebra basics: Vectors, matrices, matrix multiplication
• Calculus: Partial derivatives, chain rule
• Probability: Expectation, Bayes rule, KL divergence
• Generative Models Overview (Week 1): Basic ML concepts

1. 📖 Core Content

1.1 Intuition: Why Math Matters for Generative AI

• Linear algebra powers the transformations in neural networks — matrix multiplications, convolutions, attention mechanisms. SVD and eigendecomposition are used for PCA (preprocessing), understanding latent spaces, and analyzing optimization landscapes.
• Matrix calculus enables gradient-based learning — backpropagation is repeated application of the chain rule on matrix expressions.
• Optimization theory explains why certain training strategies work — SGD, Adam, learning rate schedules all exploit properties of the loss landscape.
• The ELBO is the central objective for VAEs, diffusion models, and many other generative models. Understanding its derivation is essential for debugging and extending these models.

1.2 Eigenvalues and Eigenvectors

1.2.1 Definition

1.2.2 Intuition

• $|\lambda| > 1$: Stretching
• $|\lambda| < 1$: Compression
• $|\lambda| = 1$: Unchanged
• $\lambda = 0$: Direction is in the nullspace

1.2.3 Eigendecomposition

• $V$: Matrix whose columns are eigenvectors
• $\Lambda$: Diagonal matrix of eigenvalues $\text{diag}(\lambda_1, ..., \lambda_n)$ For real symmetric matrices ($A = A^T$), eigenvectors are orthogonal, so $V^{-1} = V^T$:

Worked Example 1: Eigendecomposition of a 2×2 Matrix

1.3 Singular Value Decomposition (SVD)

• $U \in \mathbb{R}^{m \times m}$: Left singular vectors (orthogonal columns, $U^T U = I$)
• $\Sigma \in \mathbb{R}^{m \times n}$: Diagonal matrix of singular values $\sigma_1 \geq \sigma_2 \geq ... \geq \sigma_r > 0$
• $V \in \mathbb{R}^{n \times n}$: Right singular vectors (orthogonal columns, $V^T V = I$)
• $r$: Rank of $A$ (number of non-zero singular values)

1.3.1 Intuition

1. $V^T$ rotates the input to align with a new coordinate system
2. $\Sigma$ scales each coordinate by $\sigma_i$
3. $U$ rotates the result to produce the output (Diagram)

1.3.2 Relationship to Eigendecomposition

• $V$ contains eigenvectors of $A^T A$
• $\sigma_i^2 = \lambda_i$ (singular values squared = eigenvalues of $A^T A$) Similarly, $U$ contains eigenvectors of $A A^T$.

Worked Example 2: SVD of a 2×2 Matrix

1.4 Matrix Calculus

1.4.1 Key Rules

Worked Example 3: Gradient of a Quadratic Form

1.4.2 Gradients of Log-Likelihood for Gaussians

• MLE for Gaussian models
• VAE training (Gaussian encoder/decoder)
• KL divergence computation in diffusion models

1.5 Optimization for Generative Models

1.5.1 Convex vs Non-Convex Optimization

• Neural network parameterizations
• Min-max objectives (GANs)
• Latent variable marginalization (VAEs)

1.5.2 Stochastic Gradient Descent (SGD)

• Efficient for large datasets (compute gradient on subset)
• Noise helps escape shallow local minima
• Generalizes better than full-batch gradient descent

1.5.3 Adam Optimizer

• $\beta_1 = 0.9, \beta_2 = 0.999, \eta = 1e-4$ (GANs), $3e-4$ (diffusion models)
• Lower learning rates for discriminator/critic (two-timescale update rule)

1.6 The ELBO: Full Derivation

1.6.1 The Problem

1.6.2 Introducing the Variational Distribution

1.6.3 Derivation Step 1: Start with Log-Likelihood

1.6.4 Derivation Step 2: Multiply by 1 = q/q

1.6.5 Derivation Step 3: Apply Jensen's Inequality

1.6.6 Derivation Step 4: Rearrange the ELBO

1.6.7 Derivation Step 5: The Gap

1.6.8 The Reparameterization Trick

1.7 Why This Matters

• SVD: PCA for data preprocessing, analyzing latent space structure, low-rank approximations (LoRA) for fine-tuning
• Eigendecomposition: Understanding covariance structure of features, FID computation
• Matrix calculus: Every gradient-based training loop
• Optimization: Choosing optimizers (Adam vs SGD), learning rate schedules, warmup
• ELBO: Foundations of VAEs, diffusion models (which can be seen as hierarchical VAEs)

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Confusing SVD and Eigendecomposition

Pitfall 2: Forgetting the Reparameterization Trick Isn't Always Applicable

Pitfall 3: Ignoring the KL Vanishing Problem

Pitfall 4: Using SGD when Adam is Needed

Pitfall 5: Mishandling Matrix Calculus Gradients

• $f$ is scalar
• $W$ is $d_{out} \times d_{in}$
• $\frac{\partial f}{\partial W} = 2(Wx - y)x^T$ (check: $d_{out} \times d_{in}$ ✓)

4. 📝 Practice Questions

**Q1: Compute the eigenvalues of

and explain what they mean geometrically.**

Step 1: $\det(A - \lambda I) = (2-\lambda)^2 - 1 = \lambda^2 - 4\lambda + 3 = 0$

Step 2: $\lambda = \frac{4 \pm \sqrt{16 - 12}}{2} = \frac{4 \pm 2}{2}$

Step 3: $\lambda_1 = 3$, $\lambda_2 = 1$

Geometric interpretation: $A$ stretches space by a factor of 3 along eigenvectors corresponding to $\lambda_1$ and by 1 (no stretch) along eigenvectors corresponding to $\lambda_2$.

Eigenvectors:

• $\lambda_1 = 3$: $v_1 = [1, 1]^T/\sqrt{2}$ (direction of equal increase)
• $\lambda_2 = 1$: $v_2 = [1, -1]^T/\sqrt{2}$ (direction of difference)

**Q2: Compute the SVD of

.**

Step 1: $A$ is $2 \times 3$, so $U$ is $2 \times 2$, $\Sigma$ is $2 \times 3$, $V$ is $3 \times 3$.

Step 2:

, eigenvalues: $\lambda_1 = 4$, $\lambda_2 = 1$.

Singular values: $\sigma_1 = 2$, $\sigma_2 = 1$.

Step 3: $U$ from eigenvectors of $AA^T$: $u_1 = [0, 1]^T$, $u_2 = [1, 0]^T$.

Step 4: $V$ from $V = A^T U \Sigma^{-1}$ (more carefully):

$\Sigma = [\text{diag}(2, 1), 0_{2 \times 1}]$

$v_3$ is in the nullspace of $A$ (since $A v_3 = 0$): $v_3 = [0, 0, 1]^T$.

Step 5: Verify:

Q3: Compute $\nabla_x f$ for $f(x) = \|Ax - b\|_2^2$.

Step 1: Expand: $f = (Ax - b)^T (Ax - b) = x^T A^T A x - 2b^T A x + b^T b$

Step 2: Differentiate each term:

• $\frac{\partial}{\partial x} (x^T A^T A x) = 2 A^T A x$
• $\frac{\partial}{\partial x} (-2b^T A x) = -2 A^T b$
• $\frac{\partial}{\partial x} (b^T b) = 0$

Step 3: $\nabla_x f = 2 A^T A x - 2 A^T b = 2 A^T (Ax - b)$ Q4: Starting from $\log p(x)$, derive the ELBO and show that optimizing it maximizes a lower bound on the log-likelihood.

The derivation is in Section 1.6 above. The key steps:

1. $\log p(x) = \log \int p(x|z) p(z) dz$
2. Introduce $q(z|x)$: $\log p(x) = \log \int q(z|x) \frac{p(x|z)p(z)}{q(z|x)} dz$
3. Jensen's inequality: $\log \mathbb{E}_q[f] \geq \mathbb{E}_q[\log f]$
4. Result: $\log p(x) \geq \mathbb{E}_q[\log p(x|z)] - D_{KL}(q(z|x) \| p(z)) = \text{ELBO}$

Optimizing $\theta, \phi$ to maximize the ELBO:

• Increases the log-likelihood (by at least as much as the ELBO increases)
• Decreases $D_{KL}(q_\phi(z|x) \| p_\theta(z|x))$ (the approximation gap)

The tightness depends on how well $q_\phi(z|x)$ approximates the true posterior $p_\theta(z|x)$. Q5: In the reparameterization trick, why can't we just differentiate through the sampling operation directly?

The sampling operation $z \sim q_\phi(z|x)$ is a stochastic process. The gradient of this operation w.r.t. $\phi$ would require differentiating through a random number generator, which is non-differentiable.

Concretely, $z = \text{sample}(\mu_\phi, \sigma_\phi)$ involves calling a random number generator to produce $\epsilon \sim \mathcal{N}(0, 1)$, then computing $z = \mu_\phi + \sigma_\phi \epsilon$. The RNG call has no gradient (the operation is discrete — a number is drawn; there's no continuous path for gradient flow).

The reparameterization trick moves the stochasticity to an independent noise source $\epsilon$:

$z = \mu_\phi(x) + \sigma_\phi(x) \cdot \epsilon, \quad \epsilon \sim \mathcal{N}(0, 1)$

Now:

• The gradient can flow through $\mu_\phi$ and $\sigma_\phi$ (differentiable operations)
• $\epsilon$ is independent of $\phi$, so no gradient through sampling
• The expectation is still valid because $\mathcal{N}(z|\mu, \sigma^2) dz = \mathcal{N}(\epsilon|0, 1) d\epsilon$ Q6: For a covariance matrix $\Sigma$, explain why $\Sigma$ must be positive semidefinite and how eigendecomposition reveals this.

A covariance matrix must be positive semidefinite (PSD) because for any vector $v$:

$v^T \Sigma v = v^T \mathbb{E}[(X - \mu)(X - \mu)^T] v = \mathbb{E}[v^T (X - \mu)(X - \mu)^T v] = \mathbb{E}[(v^T (X - \mu))^2] \geq 0$

The variance of any linear combination is non-negative.

By eigendecomposition, $\Sigma = V \Lambda V^T$. PSD means all eigenvalues $\lambda_i \geq 0$. This is evident because:

• If any $\lambda_i < 0$, then $v_i^T \Sigma v_i = \lambda_i < 0$ (contradiction)
• The singular values from SVD of the data matrix are $\sqrt{\lambda_i}$, confirming non-negativity

This PSD property ensures:

• FID computation (matrix square root) is valid
• KL divergence between Gaussians is well-defined
• Cholesky decomposition for efficient sampling exists Q7: A VAE uses Adam with default parameters ($\eta=0.001$, $\beta_1=0.9$, $\beta_2=0.999$). After 100 epochs, the KL term is 0.01 (near zero). Diagnose the problem and propose a fix.

Diagnosis: KL vanishing. The model is ignoring the latent variable $z$ and functioning as an autoencoder with no regularization. The decoder is powerful enough to reconstruct $x$ without $z$, so the KL term forces $q(z|x) \to p(z) = \mathcal{N}(0, I)$, giving $D_{KL} \approx 0$.

Fixes (in order of recommendation):

1. KL annealing: Start with $\beta = 0$ in $\mathcal{L} = \text{Recon} - \beta \cdot KL$, gradually increase $\beta$ from 0 to 1 over training. This lets the model learn meaningful latent representations before the KL penalty kicks in.
2. Free bits: Set a minimum KL per dimension: $KL_{\text{min}} = \max(\beta \cdot KL, \lambda)$. This ensures each dimension carries at least $\lambda$ nats of information.
3. Reduce decoder capacity: A weaker decoder (fewer layers, smaller hidden size) forces the model to use the latent code.
4. Increase latent dimension: More dimensions give the model more capacity to encode information before the KL penalty forces them to the prior.

Q8: Show that $\nabla_\theta \mathbb{E}_{x \sim p_\theta}[f(x)] = \mathbb{E}_{x \sim p_\theta}[f(x) \nabla_\theta \log p_\theta(x)]$ (the REINFORCE/score function gradient).

Step 1: Write the expectation as an integral:

$\nabla_\theta \mathbb{E}_{x \sim p_\theta}[f(x)] = \nabla_\theta \int f(x) p_\theta(x) dx$

Step 2: Swap gradient and integral (under smoothness conditions):

$= \int f(x) \nabla_\theta p_\theta(x) dx$

Step 3: Use the log-derivative trick: $\nabla_\theta p_\theta(x) = p_\theta(x) \nabla_\theta \log p_\theta(x)$

$= \int f(x) p_\theta(x) \nabla_\theta \log p_\theta(x) dx$

Step 4: Recognize as an expectation:

$= \mathbb{E}_{x \sim p_\theta}[f(x) \nabla_\theta \log p_\theta(x)]$

This is the REINFORCE gradient estimator or score function estimator. It allows gradient estimation through non-differentiable sampling by using $\log p_\theta(x)$ instead of $\nabla_\theta x$.

Comparison with reparameterization:

• Reparameterization: Lower variance, but requires continuous $x$ and differentiable $f$
• REINFORCE: Higher variance, but works for discrete $x$ and non-differentiable $f$ Q9: For a multivariate Gaussian $\mathcal{N}(\mu, \Sigma)$, derive the gradient of the log-likelihood with respect to $\mu$ for a single sample $x$.

Step 1: Write the log-likelihood:

$\log p(x|\mu, \Sigma) = -\frac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu) - \frac{1}{2}\log|\Sigma| - \frac{D}{2}\log(2\pi)$

Step 2: Only the first term depends on $\mu$:

$\nabla_\mu \log p(x|\mu, \Sigma) = -\frac{1}{2} \nabla_\mu [(x-\mu)^T \Sigma^{-1} (x-\mu)]$

Step 3: Let $y = x - \mu$. Then $\frac{\partial y}{\partial \mu} = -I$.

Step 4: $\frac{\partial}{\partial y} (y^T \Sigma^{-1} y) = 2\Sigma^{-1} y$ (since $\Sigma^{-1}$ is symmetric)

Step 5: Chain rule:

$\nabla_\mu \log p = -\frac{1}{2} \cdot \frac{\partial (y^T \Sigma^{-1} y)}{\partial y} \cdot \frac{\partial y}{\partial \mu} = -\frac{1}{2} \cdot (2\Sigma^{-1} y)^T \cdot (-I) = y^T \Sigma^{-1} = (x-\mu)^T \Sigma^{-1}$

In column-vector convention: $\nabla_\mu \log p = \Sigma^{-1} (x - \mu)$.

The MLE for $\mu$ sets this gradient to zero: $\Sigma^{-1} (\hat{\mu} - x_{\text{avg}}) = 0 \implies \hat{\mu} = \frac{1}{N}\sum x_i$, confirming that the MLE for the mean is the sample mean. Q10: Explain why Adam's bias correction is important in the first few iterations of VAE training.

Without bias correction, Adam's moment estimates are biased toward zero because they're initialized as $m_0 = 0, v_0 = 0$:

$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$

In early iterations, $m_t$ is "warmed up" from 0. For $\beta_1 = 0.9$:

• $t=1$: $m_1 = 0.9 \cdot 0 + 0.1 \cdot g_1 = 0.1 g_1$ (10% of true gradient)
• $t=5$: $m_5 \approx 0.41 g_{\text{avg}}$ (still biased)
• $t=20$: $m_{20} \approx 0.88 g_{\text{avg}}$ (approaching true)

Bias correction divides by $1 - \beta_1^t$:

$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$

• $t=1$: $\hat{m}_1 = 0.1 g_1 / (1 - 0.9) = g_1$ ✓
• $t=5$: $\hat{m}_5 \approx 0.41 g_{\text{avg}} / (1 - 0.9^5) = 0.41 / 0.41 \approx g_{\text{avg}}$ ✓

Without bias correction, the first steps would be too small, slowing convergence — particularly noticeable in VAE training where initial gradients guide the model toward meaningful latent representations. Q11: Derive the gradient of $D_{KL}(q_\phi(z|x) \| p(z))$ with respect to $\phi$ where $q_\phi(z|x) = \mathcal{N}(\mu_\phi, \sigma_\phi^2)$ and $p(z) = \mathcal{N}(0, 1)$.

Step 1: Write the KL in closed form for Gaussians:

$D_{KL}(\mathcal{N}(\mu, \sigma^2) \| \mathcal{N}(0, 1)) = \log\frac{1}{\sigma} + \frac{\sigma^2 + \mu^2}{2} - \frac{1}{2}$

$= -\log\sigma + \frac{\sigma^2 + \mu^2}{2} - \frac{1}{2}$

Step 2: Differentiate w.r.t. $\mu$:

$\frac{\partial KL}{\partial \mu} = \mu$

Step 3: Differentiate w.r.t. $\sigma$ (using $\sigma$ as the standard deviation):

$\frac{\partial KL}{\partial \sigma} = -\frac{1}{\sigma} + \sigma = \sigma - \frac{1}{\sigma} = \frac{\sigma^2 - 1}{\sigma}$

Step 4: In practice, we parameterize $\log\sigma^2$ (log-variance) for numerical stability. Let $s = \log\sigma^2$, so $\sigma = e^{s/2}$:

$\frac{\partial KL}{\partial s} = \frac{\partial KL}{\partial \sigma} \cdot \frac{\partial \sigma}{\partial s} = \left(\sigma - \frac{1}{\sigma}\right) \cdot \frac{e^{s/2}}{2} = \left(e^{s/2} - e^{-s/2}\right) \cdot \frac{e^{s/2}}{2}$

$= \frac{e^s - 1}{2} = \frac{\sigma^2 - 1}{2}$

The gradient tells us:

• If $\mu > 0$, push $\mu$ toward zero (reduce mean)
• If $\sigma > 1$, push $\sigma$ toward 1 (reduce variance)
• If $\sigma < 1$, push $\sigma$ toward 1 (increase variance)

The KL pushes the posterior toward a unit Gaussian — neither too wide nor too narrow. Q12: In diffusion models, the training loss $\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$ resembles a denoising objective. Show how this relates to the ELBO.

The diffusion model's variational bound can be written as:

$\mathcal{L}_{vb} = \sum_{t=1}^T \mathbb{E}_{x_0, \epsilon_t}[D_{KL}(q(x_{t-1}|x_t, x_0) \| p_\theta(x_{t-1}|x_t))]$

Each KL term is between two Gaussians:

• $q(x_{t-1}|x_t, x_0) = \mathcal{N}(\tilde{\mu}_t(x_t, x_0), \tilde{\beta}_t I)$ (the forward posterior, known)
• $p_\theta(x_{t-1}|x_t) = \mathcal{N}(\mu_\theta(x_t, t), \sigma_t^2 I)$ (the model, learned)

Using the closed-form KL for Gaussians:

$D_{KL}(q \| p_\theta) = \frac{1}{2\sigma_t^2} \|\tilde{\mu}_t(x_t, x_0) - \mu_\theta(x_t, t)\|^2 + \text{const}$

We parameterize $\mu_\theta$ as:

$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}} \left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t)\right)$

And the forward posterior mean is:

$\tilde{\mu}_t(x_t, x_0) = \frac{1}{\sqrt{\alpha_t}} \left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_t\right)$

Substituting, the KL simplifies to:

$D_{KL}(q \| p_\theta) \propto \|\epsilon_t - \epsilon_\theta(x_t, t)\|^2$

Thus, the diffusion training loss $\|\epsilon - \epsilon_\theta(x_t, t)\|^2$ is equivalent to minimizing the KL divergence between the true denoising step and the model's prediction at each timestep — which is exactly the ELBO for diffusion models. This demonstrates the deep connection between denoising score matching and variational inference.