Autoregressive Models: PixelCNN, PixelRNN, and Sequential Generation

🎯 Learning Objectives

• Understand the autoregressive factorization of the data likelihood
• Implement causal/masked convolutions for efficient image generation
• Compare PixelRNN (sequential) and PixelCNN (parallel) architectures
• Explain how autoregressive models handle long-range dependencies
• Analyze the trade-offs between autoregressive and latent variable models

📋 Prerequisites

• Convolutional Neural Networks: Basic CNN operations
• Sequence Models (optional): RNN/LSTM basics
• Probability: Chain rule of probability, conditional distributions
• Generative Models Overview (Week 1): Taxonomy of generative models

1. 📖 Core Content

1.1 Intuition: What is an Autoregressive Generative Model?

1. Provide tractable exact likelihoods (no approximation or lower bound needed)
2. Have stable training (just maximum likelihood, no adversarial training)
3. Achieve state-of-the-art density estimation on many benchmarks
4. Power large language models (GPT, LLaMA) — the dominant paradigm in NLP The downside: sequential generation is slow — for a 256×256 image, you must generate 65,536 pixels one at a time, each requiring a forward pass through a neural network.

1.2 The Autoregressive Factorization

Worked Example 1: Autoregressive Modeling of Binary Sequences

• $p(x_1=1) = 0/4 = 0$, $p(x_1=0) = 4/4 = 1$
• $p(x_2=1|x_1=0) = 2/4 = 0.5$, $p(x_2=0|x_1=0) = 2/4 = 0.5$
• $p(x_3=1|x_1=0,x_2=0) = 2/4 = 0.5$, $p(x_3=0|x_1=0,x_2=0) = 2/4 = 0.5$
• $p(x_4=1|x_1=0,x_2=0,x_3=0) = 1/2 = 0.5$, $p(x_4=0|x_1=0,x_2=0,x_3=0) = 1/2 = 0.5$
• $p(x_4=1|x_1=0,x_2=0,x_3=1) = 1/1 = 1$, $p(x_4=0|x_1=0,x_2=0,x_3=1) = 0$ Step 3: Compute likelihood of a sequence. For [0,0,0,0]:

1. Sample $x_1 \sim p(x_1)$ → always 0
2. Sample $x_2 \sim p(x_2|x_1)$ → 0 or 1 equally likely
3. Sample $x_3 \sim p(x_3|x_1,x_2)$ → 0 or 1 equally likely
4. Sample $x_4 \sim p(x_4|x_1,x_2,x_3)$ → depends on previous values This small example illustrates the core mechanism: sequential factorization with conditional distributions.

1.3 PixelRNN: Sequential Processing with Recurrent Networks

• Row LSTM: Process entire rows at once using a convolutional LSTM
• Diagonal BiLSTM: Process along diagonals for better coverage Despite improvements, PixelRNN remains too slow for practical image generation at high resolutions.

1.4 PixelCNN: Parallelizable with Masked Convolutions

1.4.1 The Key Insight

1.4.2 Mask Type A and Type B

1.4.3 Implementation

python
Run Inline
# runnable
import torch
import torch.nn as nn
import torch.nn.functional as F
class MaskedConv2d(nn.Module):
    def __init__(self, mask_type, in_channels, out_channels, kernel_size=3):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, padding=kernel_size//2)
        self.register_buffer('mask', self.conv.weight.data.clone())
        _, _, h, w = self.mask.size()
        self.mask.fill_(1)
        # Center of kernel
        center_y, center_x = h // 2, w // 2
        if mask_type == 'A':
            # Center pixel is masked (for first layer)
            self.mask[:, :, center_y, center_x] = 0
        # Mask out future pixels (below center row)
        self.mask[:, :, center_y+1:, :] = 0
        # Mask out future pixels (same row, to the right of center)
        self.mask[:, :, center_y, center_x+1:] = 0
    def forward(self, x):
        self.conv.weight.data *= self.mask
        return self.conv(x)

1.4.4 PixelCNN Architecture

• Stack of masked convolutions: Typically 10-20 layers to build a large receptive field
• Gated activations: $y = \tanh(W_f * x) \odot \sigma(W_g * x)$ improves performance
• Residual connections: Help train deeper networks
• 256-way softmax: Pixel values (0-255) treated as categorical, not continuous

Worked Example 2: Receptive Field Growth in PixelCNN

Worked Example 3: Computing the Loss for Autoregressive Models

1.5 Gated PixelCNN and PixelCNN++

Gated PixelCNN

• Vertical stack: Sees all pixels above the current row (captures context)
• Horizontal stack: Sees pixels to the left on the same row (captures local detail) The two stacks are combined via gating:

PixelCNN++

1. Discretized logistic mixture instead of 256-way softmax (better likelihood, fewer outputs)
2. Downsampling for larger receptive fields
3. Dropout for regularization
4. Multi-scale architecture: Generate at low resolution first, then upsample with conditioning The discretized logistic mixture loss models each pixel's value as a mixture of logistic distributions, discretized to 256 levels:

1.6 Autoregressive Models Beyond Images

1.7 Edge Cases & Gotchas

• Pixel ordering matters: Raster scan is arbitrary — different orderings produce different models. Some orderings (e.g., S-curve) capture different dependencies.
• Blind spots: Standard PixelCNN can't see pixels directly above due to masking. This creates a "blind spot" in the receptive field.
• Training/Sampling mismatch: During training, the model sees ground-truth context. During sampling, it sees its own predictions, which may have different statistics (exposure bias).
• High computational cost: Each pixel generation requires a full forward pass, making high-resolution autoregessive generation impractical.
• Color channels: For RGB images, pixels within the same location must also be ordered (e.g., R→G→B).

1.8 Why This Matters

• Week 4 (Diffusion Models): Diffusion offers faster sampling for images, but autoregressive models provide better density estimation
• BSDA5004 (LLMs): Autoregressive transformers are the dominant paradigm in NLP
• Practical trade-off: Use autoregressive for text (where sequential generation is natural), use diffusion for images (where parallel generation matters) Autoregressive image models like PixelCNN++ were state-of-the-art before diffusion models surpassed them, but the core principle (chain rule factorization) remains fundamental to generative modeling.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Confusing Autoregressive Models with RNNs

Pitfall 2: Forgetting That Training and Sampling Have Different Computation Graphs

Pitfall 3: Ignoring the Blind Spot in PixelCNN

Pitfall 4: Treating Pixel Values as Continuous Instead of Categorical

4. 📝 Practice Questions

Q1: Why are autoregressive models considered "exact likelihood" models while VAEs are not?

Autoregressive models use the chain rule: $p(x) = \prod p(x_i|x_{<i})$, which is mathematically exact — no approximations. Each conditional is directly modeled and the joint is the product.

VAEs introduce a latent variable $z$: $p(x) = \int p(x|z)p(z) dz$. This integral is intractable, so VAEs optimize a lower bound (ELBO) instead of the exact likelihood. The gap between the ELBO and the true log-likelihood is the KL divergence between the approximate posterior $q(z|x)$ and the true posterior $p(z|x)$, which is generally > 0.

Autoregressive models pay for exactness with sequential sampling; VAEs pay with approximate likelihood but gain fast parallel sampling. Q2: For a 64×64 grayscale image, how many sequential neural network evaluations are needed to generate one image with (a) PixelCNN, (b) DDPM with 1000 steps, (c) GAN?

(a) PixelCNN: $64 \times 64 = 4096$ sequential evaluations (one per pixel). (b) DDPM: 1000 sequential evaluations (one per denoising step). (c) GAN: 1 evaluation (generator transforms noise $z$ to image in one forward pass).

Note: PixelCNN's per-pixel evaluations can be sped up with caching (reusing hidden states), but still requires $O(N)$ steps. This comparison highlights why autoregressive models are impractical for high-resolution image generation compared to diffusion models or GANs. Q3: Explain why Mask A (center pixel excluded) is necessary in the first layer of PixelCNN.

If the center pixel were visible in the first layer, the model could trivially copy the input pixel value to the output. The first layer convolution at position (i,j) would receive the pixel value $x_{i,j}$ as input and could simply pass it through. This would allow the model to predict $p(x_{i,j} | x_{<i,j})$ with $x_{i,j}$ itself in the conditioning set, which is a form of "label leakage" — the model would learn the identity mapping and never learn true conditional distributions.

Mask A prevents this by zeroing out the center of the convolution kernel, ensuring the first layer's representation of pixel (i,j) depends only on neighbors before it in raster order. Mask B in subsequent layers can include the center because the information flowing through is already processed (not raw pixel values). Q4: Why does PixelCNN++ use a discretized logistic mixture instead of 256-way softmax?

1. Parameter efficiency: A mixture of $K$ logistics requires $3K$ parameters per pixel ($K$ mixing weights, $K$ means, $K$ scales). With $K=5$, that's 15 parameters. The 256-way softmax requires 256 parameters per pixel. 15 < 256.
2. Smoothness: The logistic distribution naturally imposes smoothness over pixel intensities — nearby values (e.g., 127 and 128) should have similar probabilities. The softmax treats each level independently.
3. Multi-modality: A mixture of logistics can represent complex, multi-modal pixel distributions (e.g., a pixel that is either very dark or very bright) better than a single unimodal distribution.

The discretized logistic is obtained by integrating the logistic CDF over each integer bin: $P(x=v) = F(v+0.5 | \mu, s) - F(v-0.5 | \mu, s)$. Q5: Design a simple autoregressive model for a 4×4 binary image (each pixel is 0 or 1). How many parameters would a full conditional model (no weight sharing) have?

The number of possible previous pixel configurations for position $i$ is $2^{i-1}$. For $i=16$ (last pixel), there are $2^{15} = 32768$ possible contexts. A full table would have:

$2^0 + 2^1 + 2^2 + ... + 2^{15} = 2^{16} - 1 = 65535 \text{ entries}$

Each entry stores a single Bernoulli probability $p(x_i=1|x_{<i})$, so 65535 parameters. This demonstrates why weight sharing (via neural networks) is essential — the naive table approach is exponential in the number of pixels.

For comparison, a simple PixelCNN with 4 masked conv layers (8 filters each) would have roughly $4 \times (9 \times 8^2) \approx 2304$ parameters, a fraction of the 65535 needed for the full table. Q6: GPT is an autoregressive text model. How does it differ from PixelCNN in terms of (a) the factorization, (b) the conditioning architecture, (c) the output distribution?

(a) Factorization: Both use the chain rule. GPT: $p(\text{token}_1, ..., \text{token}_T) = \prod p(\text{token}_t | \text{token}_{<t})$. PixelCNN: $p(\text{pixel}_1, ..., \text{pixel}_N) = \prod p(\text{pixel}_i | \text{pixel}_{<i})$. The concept is identical.

(b) Conditioning architecture: GPT uses Transformer decoder blocks with causal self-attention (each token attends to all previous tokens). PixelCNN uses masked convolutions. Transformers have larger receptive fields (any token can attend to any previous token) while convolutions have limited local receptive fields.

(c) Output distribution: GPT uses a softmax over the vocabulary (typically 50K+ tokens). PixelCNN uses a 256-way softmax or discretized logistic mixture over pixel intensities.

Despite different architectures, both implement the same core idea: sequential next-element prediction. Q7: For a PixelCNN with 12 layers of 3×3 masked convolutions (no dilation), what's the receptive field of the final layer?

With $L$ layers of $k \times k$ convolutions (no dilation), the receptive field is:

$RF = 1 + L \cdot (k - 1)$

For $L=12$, $k=3$: $RF = 1 + 12 \cdot 2 = 25$.

The receptive field is $25 \times 25$ pixels. For a 64×64 image, this covers about 15% of the image. For a 256×256 image, only about 1% of the image is visible.

This limited receptive field is why PixelCNN struggles with global structure — it cannot see far enough. Dilation or downsampling (as in PixelCNN++) is needed for larger images. Q8: What is "teacher forcing" in the context of autoregressive models? Why is it both helpful and harmful?

Teacher forcing means during training, the model receives ground-truth context $x_{<i}$ to predict $x_i$, but during generation, it receives its own previously-generated samples as context.

Helpful because: Training is parallelizable (all $N$ conditionals can be computed simultaneously), gradients are well-behaved (no sampling noise), and training converges faster.

Harmful because: There's a mismatch between training and inference (exposure bias). A small error during generation can compound — if the model generates $x_2$ incorrectly, then $x_3$ is conditioned on a wrong context, leading to cascading errors. This is why techniques like scheduled sampling (mixing ground-truth and generated context during training) are sometimes used. Q9: In PixelCNN, why is it important to have a separate "vertical stack" and "horizontal stack" (Gated PixelCNN)?

A single masked convolution has a blind spot: pixels directly above the current position (in the same column) are masked out. This happens because:

• Future pixels (below center row) are masked
• Future pixels on the same row (right of center) are masked
• But pixels ABOVE the center row are NOT masked (they're past context)
• However, the single mask zeros out the entire top-right quadrant and the center-right part of the middle row, which includes some pixels above

The vertical stack handles ALL pixels above the current row, while the horizontal stack handles pixels to the left on the same row. The two stacks process information independently and are combined via gating. This split design eliminates the blind spot entirely. Q10: How does an autoregressive model handle RGB color channels at a single pixel location?

For RGB images, the three color channels at a single location (i,j) must be ordered. The standard approach (used in PixelCNN) is to process them as R → G → B within each pixel location.

The factorization becomes:

$p(x_{i,j,R}, x_{i,j,G}, x_{i,j,B} | x_{<(i,j)}) = p(x_{i,j,R} | x_{<(i,j)}) \cdot p(x_{i,j,G} | x_{<(i,j)}, x_{i,j,R}) \cdot p(x_{i,j,B} | x_{<(i,j)}, x_{i,j,R}, x_{i,j,G})$

This means the model first predicts the red channel (conditioned on all previous pixels), then green (conditioned on previous pixels + red at current pixel), then blue (conditioned on previous pixels + red + green at current pixel).

Architecturally, this is achieved by having separate output heads for R, G, B, where G's head receives R's output and B's head receives R and G's outputs. Q11: Compare autoregressive models and normalizing flows as exact likelihood methods.

Normalizing flows offer parallel sampling (sample $z \sim \mathcal{N}(0, I)$, transform to $x$ in one pass) while maintaining exact likelihood. However, flows require designing invertible transformations with tractable Jacobians, which constrains architecture choices. Autoregressive models have more architectural freedom but pay for it with sequential sampling.

In practice, flows are rarely used for high-resolution images because the invertibility constraint makes them parameter-inefficient compared to autoregressive or diffusion models. Q12: Given PixelCNN's slow sampling, propose a hybrid approach that combines autoregressive models with another generative model type.

Hybrid: Autoregressive prior + Diffusion decoder

• High-level structure: Use an autoregressive model to generate a low-resolution (e.g., 8×8) image
• Super-resolution: Use a diffusion model (or upsampling network) conditioned on the low-res output to generate the full-resolution (256×256) image

This leverages:

• Autoregressive strength: Excellent density estimation for low-res (few pixels, manageable sequential cost)
• Diffusion strength: Fast, high-quality upsampling (only 20-50 steps in pixel space)

Another approach: Autoregressive latent model (e.g., VQ-VAE + PixelCNN):

1. Compress the image to discrete latent codes via VQ-VAE (e.g., 32×32 grid of codes)
2. Use PixelCNN on the latent grid (1024 sequential steps instead of 65536)
3. Decode the latent grid back to pixels in one forward pass

This is the basis of models like DALL·E, VQ-GAN, and Parti — combining the best of both worlds.

5. 🔗 Cross-References

• Previous: DDIMs (Week 5) — Accelerated diffusion sampling
• Next: Conditional Generation (Week 7) — Adding conditioning to generative models
• Related: Evaluation Metrics (Week 8) — How to measure autoregressive model quality
• External: Van den Oord et al., "Pixel Recurrent Neural Networks" (ICML 2016)
• External: Van den Oord et al., "Conditional Image Generation with PixelCNN Decoders" (NIPS 2016)
• External: Salimans et al., "PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood" (ICLR 2017) Join Discord PreviousDDIMsNextConditional Generation