GPT Architecture: Decoder-Only Transformer & Autoregressive Generation

🎯 Learning Objectives

• Understand GPT's decoder-only architecture and why it's effective for generation
• Explain autoregressive generation and implement decoding strategies
• Compare greedy search, beam search, top-k, and top-p sampling
• Analyze the scaling behavior of GPT models

📋 Prerequisites

• Decoder layer and causal masking (Week 2)
• Causal Language Modeling objective (Week 3)
• Tokenization basics

1. 📖 Core Content

1.1 From Encoder-Decoder to Decoder-Only

• Pre-train on the CLM objective (next token prediction)
• Be fine-tuned for diverse downstream tasks
• Generate coherent text autoregressively (Diagram) How GPT differs from the original Transformer decoder:

1. No cross-attention: There's no encoder to attend to
2. Only one attention type: Masked self-attention throughout
3. Learned positional embeddings: Not sinusoidal
4. Pre-LN: LayerNorm before sublayers
5. Larger feed-forward dimension: Typically 4× d_model

1.2 Autoregressive Generation

How Generation Works

1. Feed the sequence $x_1, ..., x_{t-1}$ into the model
2. Get probability distribution over vocabulary: $P(x_t | x_{<t})$
3. Sample/select the next token
4. Append to sequence and repeat

Formal Definition

1.3 Decoding Strategies

Greedy Decoding

Beam Search

1. Start with k candidates (beams)
2. At each step, expand each beam to all possible tokens
3. Keep top k sequences by cumulative probability
4. Continue until all beams hit [EOS] or max length

python
Run Inline
# runnable
import numpy as np
def beam_search(model, prompt, beam_width=3, max_len=20, vocab_size=100):
    """
    Simplified beam search for text generation
    Args:
        model: Function that takes token IDs and returns logits
        prompt: List of initial token IDs
        beam_width: Number of beams to maintain
        max_len: Maximum generation length
        vocab_size: Size of vocabulary
    Returns:
        best_sequence: The most probable generated sequence
    """
    # Initialize beams: each is (sequence, log_probability)
    beams = [(prompt, 0.0)]
    completed = []
    for step in range(max_len):
        candidates = []
        for seq, log_prob in beams:
            if len(seq) > len(prompt) and seq[-1] == 1:  # [EOS] token = 1
                completed.append((seq, log_prob))
                continue
            # Get model prediction
            logits = model(seq)  # Shape: (vocab_size,)
            probs = np.exp(logits - np.max(logits))
            probs = probs / np.sum(probs)
            # Top-k tokens
            top_k = np.argsort(probs)[-beam_width:]
            for token in top_k:
                new_seq = seq + [token]
                new_log_prob = log_prob + np.log(probs[token] + 1e-10)
                candidates.append((new_seq, new_log_prob))
        if not candidates:
            break
        # Keep top beam_width candidates
        candidates.sort(key=lambda x: x[1], reverse=True)
        beams = candidates[:beam_width]
    # Return best sequence (or best among completed)
    if completed:
        completed.sort(key=lambda x: x[1], reverse=True)
        return completed[0][0]
    return beams[0][0] if beams else prompt
# Dummy model for demonstration
def dummy_model(tokens):
    """Mock model returning random logits"""
    np.random.seed(len(tokens))
    return np.random.randn(100)
prompt = [5, 12]  # "The cat"
result = beam_search(dummy_model, prompt, beam_width=3, max_len=5)
print(f"Prompt: {prompt}")
print(f"Generated: {result}")

Top-K Sampling

1. Get probability distribution
2. Find the k-th highest probability
3. Zero out all probabilities below this threshold
4. Renormalize and sample

python
Run Inline
# runnable
import numpy as np
def top_k_sampling(logits, k=50, temperature=1.0):
    """
    Top-K sampling
    Args:
        logits: Raw logits from model (vocab_size,)
        k: Number of top tokens to consider
        temperature: Higher = more random, lower = more greedy
    Returns:
        sampled_token: Selected token ID
    """
    # Apply temperature
    logits = logits / temperature
    # Find top-k threshold
    top_k_values = np.sort(logits)[-k]
    top_k_threshold = top_k_values
    # Zero out below threshold
    filtered_logits = np.where(logits >= top_k_threshold, logits, -np.inf)
    # Convert to probabilities and sample
    probs = np.exp(filtered_logits - np.max(filtered_logits))
    probs = probs / np.sum(probs)
    sampled_token = np.random.choice(len(logits), p=probs)
    return sampled_token
# Example
np.random.seed(42)
logits = np.random.randn(1000)
token = top_k_sampling(logits, k=50, temperature=0.8)
print(f"Sampled token: {token}")
# Compare temperatures
for temp in [0.1, 0.8, 2.0]:
    tokens = [top_k_sampling(logits, k=50, temperature=temp) for _ in range(10)]
    print(f"Temp {temp}: {tokens[:5]}...")

Top-P (Nucleus) Sampling

1. Sort tokens by probability descending
2. Add tokens to the set until cumulative probability ≥ p
3. Renormalize and sample from this set

python
Run Inline
# runnable
import numpy as np
def top_p_sampling(logits, p=0.9, temperature=1.0):
    """
    Top-P (Nucleus) sampling
    Args:
        logits: Raw logits (vocab_size,)
        p: Cumulative probability threshold
        temperature: Higher = more random
    Returns:
        sampled_token: Selected token ID
    """
    logits = logits / temperature
    # Sort by probability
    probs = np.exp(logits - np.max(logits))
    probs = probs / np.sum(probs)
    sorted_indices = np.argsort(probs)[::-1]
    sorted_probs = probs[sorted_indices]
    # Find cutoff
    cumulative = np.cumsum(sorted_probs)
    cutoff_idx = np.searchsorted(cumulative, p) + 1
    # Keep only top-p tokens
    mask = np.zeros_like(logits, dtype=bool)
    mask[sorted_indices[:cutoff_idx]] = True
    filtered_probs = np.where(mask, probs, 0)
    filtered_probs = filtered_probs / np.sum(filtered_probs)
    sampled_token = np.random.choice(len(logits), p=filtered_probs)
    return sampled_token
# Example
np.random.seed(42)
logits = np.random.randn(1000)
token = top_p_sampling(logits, p=0.9, temperature=1.0)
print(f"Sampled token: {token}")

1.4 Temperature Scaling

• $\tau = 1$: Standard softmax
• $\tau < 1$ (e.g., 0.7): Sharper distribution, more conservative
• $\tau > 1$ (e.g., 1.5): Flatter distribution, more random
• $\tau \to 0$: Greedy (always picks max)
• $\tau \to \infty$: Uniform sampling

1.5 The GPT Family Scale

1.6 Why This Matters

• GPT-3/GPT-4: The foundation of ChatGPT
• LLaMA: Meta's open-source LLM family
• Claude: Anthropic's safety-focused models
• Mistral: Efficient open-source models All use decoder-only architectures with causal masking and autoregressive generation.

4. 📐 Key Formulas / Concepts

5. ⚠️ Common Pitfalls

Pitfall 1: Using greedy decoding for creative tasks

Pitfall 2: Confusing model size with context length

Pitfall 3: Not setting a generation length limit

6. 📝 Practice Questions

Q1: A GPT-3 model has context length 2048 and generates 10 tokens per request. If KV-cache reduces compute to O(n) per new token, how many FLOPs per token?

GPT-3: d_model=12288, n_layers=96, d_ff=49152

Per token FLOPs ≈ 2 × n_layers × (4 × d_model² + 2 × d_model × d_ff) = 2 × 96 × (4 × 12288² + 2 × 12288 × 49152) = 2 × 96 × (603,979,776 + 1,207,959,552) = 2 × 96 × 1,811,939,328 ≈ 348 billion FLOPs per token

For 10 tokens: ~3.48 trillion FLOPs for generation. Q2: Why does beam search with k=3 produce different results than greedy decoding?

Greedy makes locally optimal choices that might lead to suboptimal global sequences. Beam search maintains multiple hypotheses, potentially finding a path where an initial "suboptimal" token leads to a better overall sequence.

Example: Greedy: "The cat sat on the mat" (prob 0.7) Alternative path: "The cat rested on..." (prob 0.9 overall but "rested" was only 0.3 vs "sat" at 0.7)

Beam search would catch this; greedy wouldn't. Q3: What is the probability that top-p (p=0.9) sampling selects a token outside the top 100 most probable tokens?

If the top 100 tokens have cumulative probability ≥ 0.9, then no token outside the top 100 can be selected (since we only sample from the set of tokens whose cumulative probability reaches 0.9).

However, if the top 100 tokens have cumulative probability < 0.9 (unlikely for typical softmax distributions), then some tokens beyond rank 100 would be included.

In practice, token distributions are typically long-tailed: the top few dozen tokens capture most of the probability mass. Q4: Compare the computational cost of generating 100 tokens with beam search (k=4) vs top-k sampling.

Beam search (k=4): Each step processes 4× the tokens of a single sequence. The model must compute logits for 4 sequences at each step. A 100-token generation with beam search requires ~400 forward passes (4 beams × 100 steps) plus beam management overhead.

Top-k sampling: One forward pass per step. A 100-token generation requires 100 forward passes.

Beam search is ~4× more expensive computationally for the same output length. The trade-off is better quality for tasks like translation. Q5: If GPT-3 (175B parameters) processes the prompt "The Eiffel Tower is located in" at an average of 0.02 seconds per token on an A100, how long to generate a 100-token response?

First token: Must process the full prompt (let's say 10 tokens) + generate token 1. This uses full attention (O(n²)). Subsequent tokens: Use KV-cache, O(n) per token.

Total time ≈ 0.02 × 100 = 2 seconds (assuming KV-cache makes per-token time roughly constant).

Without KV-cache: 0.02 × (10² + 11² + 12² + ... + 110²) / 10² ≈ much longer. Q6: A model generates "I love programming" with probabilities P(I)=0.9, P(love|I)=0.7, P(programming|I,love)=0.8. What is the sequence probability?

P(I love programming) = 0.9 × 0.7 × 0.8 = 0.504

The joint probability of the three-token sequence is 50.4%. Q7: Why might top-p sampling be preferred over top-k sampling?

Top-p is adaptive: when the model is confident (one token dominates), p allows fewer tokens; when uncertain (many plausible tokens), p allows more tokens.

Top-k is fixed: k=50 allows 50 tokens regardless of how many are actually plausible. For highly predictable text (e.g., "I live in ____"), the top 2 tokens might have 99% probability, leaving 48 near-zero-probability tokens. Top-k would still include all 50, potentially selecting an unlikely token.

Top-p avoids both too-restrictive (when many tokens are plausible) and too-permissive (when few tokens are plausible) settings. Q8: What is the effect of temperature τ=0.5 on a distribution with logits [2, 1, 0, -1]?

Before temperature: P = softmax([2, 1, 0, -1]) e²=7.39, e¹=2.72, e⁰=1.00, e⁻¹=0.37 Sum = 11.48 P = [0.644, 0.237, 0.087, 0.032]

After temperature τ=0.5: logits/0.5 = [4, 2, 0, -2] e⁴=54.60, e²=7.39, e⁰=1.00, e⁻²=0.135 Sum = 63.125 P = [0.865, 0.117, 0.016, 0.002]

Lower temperature makes the distribution sharper — the highest probability token becomes even more dominant. Q9: Describe a limitation of decoder-only architectures compared to encoder-decoder for translation tasks.

In encoder-decoder models (like T5), cross-attention allows the decoder to directly "look at" the full source sentence at each generation step. This produces better alignment between source and target.

In decoder-only models, the source and target are concatenated: "Translate to French: Hello → [SEP] Bonjour". The model must attend to the source through causal attention. For very long source sentences, the source information may be "diluted" across many tokens.

This is why encoder-decoder architectures sometimes outperform decoder-only models for tasks requiring strong input-output alignment. Q10: How does repetition occur in autoregressive generation and how can it be prevented?

Repetition occurs because:

1. High-probability loops: Once the model repeats a phrase, the context contains that phrase, making it more likely to be generated again
2. Lack of global planning: Autoregressive models don't "plan" ahead

Prevention methods:

1. Repetition penalty: Reduce probability of tokens that have already appeared
2. N-gram blocking: Prevent repeating any n-gram
3. Diversity sampling: Encourage token diversity
4. Top-k/p + temperature: Already reduces repetition vs greedy
5. Presence penalty: Hyperparameter that penalizes tokens based on whether they've appeared

7. 🔗 Cross-References

• Next: BERT Architecture (Week 5)
• Previous: Pre-training Objectives
• Video: BSDA5004 Week 4 and Week 6 transcripts Join Discord PreviousPre-training ObjectivesNextBERT Architecture