Pre-training Objectives: Causal LM, Masked LM, and Beyond

🎯 Learning Objectives

• Distinguish between Causal LM and Masked LM objectives
• Understand why different architectures use different objectives
• Implement the training loop for both CLM and MLM
• Explain Next Sentence Prediction and other auxiliary objectives
• Compare autoregressive vs autoencoding models

📋 Prerequisites

• Transformer encoder-decoder architecture
• Cross-entropy loss
• Self-attention and masking

1. 📖 Core Content

1.1 What is Pre-training?

• Predicting the next word (Causal LM)
• Predicting masked/filled words (Masked LM)
• Predicting if two sentences are consecutive (NSP)

1.2 Causal Language Modeling (CLM)

Intuition

Formal Definition

Architecture: Decoder-Only

Worked Example

1.3 Masked Language Modeling (MLM)

Intuition

Formal Definition

The 15% Masking Strategy

python
Run Inline
# runnable
import numpy as np
def apply_bert_mask(tokens, mask_token_id, vocab_size, mask_prob=0.15):
    """
    Apply BERT-style masking to a token sequence
    Args:
        tokens: List of token IDs
        mask_token_id: ID of [MASK] token
        vocab_size: Size of vocabulary
        mask_prob: Probability of masking each token
    Returns:
        masked_tokens: Tokens after masking
        labels: Original tokens (with -100 for non-masked positions)
    """
    seq_len = len(tokens)
    masked_tokens = tokens.copy()
    labels = [-100] * seq_len  # -100 positions are ignored in loss
    for i in range(seq_len):
        if np.random.random() < mask_prob:
            labels[i] = tokens[i]  # Store original token for loss
            r = np.random.random()
            if r < 0.8:
                masked_tokens[i] = mask_token_id  # 80%: [MASK]
            elif r < 0.9:
                masked_tokens[i] = np.random.randint(0, vocab_size)  # 10%: random
            # else 10%: keep unchanged (labels still has original)
    return masked_tokens, labels
# Example
np.random.seed(42)
tokens = [5, 12, 8, 23, 45, 67, 89, 34, 56, 78]
mask_token_id = 0
vocab_size = 1000
masked, labels = apply_bert_mask(tokens, mask_token_id, vocab_size)
print("Original:", tokens)
print("Masked:  ", masked)
print("Labels:  ", labels)
print("Masked positions:", [i for i, l in enumerate(labels) if l != -100])

1.4 CLM vs MLM: Comparison

1.5 Next Sentence Prediction (NSP)

• Positive example: A = "The cat sat on the mat." B = "It was very comfortable." (B follows A)
• Negative example: A = "The cat sat on the mat." B = "The capital of France is Paris." (random pair)

python
Run Inline
# runnable
import numpy as np
def create_nsp_example(sentence_a_tokens, sentence_b_tokens, is_next):
    """
    Create an NSP training example
    Args:
        sentence_a_tokens: Token IDs for sentence A
        sentence_b_tokens: Token IDs for sentence B
        is_next: 1 if B follows A, 0 otherwise
    Returns:
        input_ids: [CLS] + A_tokens + [SEP] + B_tokens + [SEP]
        segment_ids: 0 for A, 1 for B
        label: is_next (1 or 0)
    """
    cls_id = 101  # [CLS]
    sep_id = 102  # [SEP]
    input_ids = [cls_id] + sentence_a_tokens + [sep_id] + sentence_b_tokens + [sep_id]
    segment_ids = [0] + [0]*len(sentence_a_tokens) + [0] + [1]*len(sentence_b_tokens) + [1]
    label = 1 if is_next else 0
    return input_ids, segment_ids, label
# Example
sent_a = [12, 34, 56]  # "The cat sat"
sent_b_pos = [78, 90, 23]  # "on the mat" (positive)
sent_b_neg = [45, 67, 89]  # "Paris is capital" (negative)
pos_ids, pos_seg, pos_label = create_nsp_example(sent_a, sent_b_pos, True)
neg_ids, neg_seg, neg_label = create_nsp_example(sent_a, sent_b_neg, False)
print(f"Positive example - Input: {pos_ids}")
print(f"Segment IDs: {pos_seg}")
print(f"Label: {pos_label}")

1.6 Other Pre-training Objectives

1.7 Why This Matters

• What the model learns: CLM excels at generation, MLM excels at understanding
• Architecture: Decoder-only for CLM, encoder-only for MLM
• Downstream performance: Different objectives suit different tasks
• Computational cost: CLM trains on all tokens, MLM only on 15%

4. 📐 Key Formulas / Concepts

5. ⚠️ Common Pitfalls

Pitfall 1: Thinking MLM and CLM are interchangeable

Pitfall 2: Confusing "pre-training" with "training from scratch"

• Pre-training: Self-supervised learning on large unlabeled data (objective: LM/MLM)
• Fine-tuning: Supervised learning on labeled downstream data (objective: task-specific)
• Training from scratch: Supervised on task data only (no pre-training) Pre-training is the expensive first stage that provides general language understanding.

Pitfall 3: Ignoring the 15% masking cost

6. 📝 Practice Questions

Q1: For a sentence of 100 tokens with 15% masking, how many tokens contribute to the MLM loss?

15 tokens contribute to the loss (100 × 0.15 = 15). The remaining 85 tokens serve only as context. This is why MLM is less data-efficient per training step than CLM (where all 100 tokens contribute).

However, MLM provides richer training signal per token because it uses bidirectional context. Q2: Why does BERT use [MASK] only 80% of the time, not 100%?

If [MASK] was used 100% of the time:

• The model would never learn to process non-masked tokens correctly
• During fine-tuning, there are no [MASK] tokens, creating a distribution mismatch
• The model would be confused by real tokens

The 80-10-10 split forces the model to maintain good representations for all tokens, not just masked ones. Q3: Calculate the CLM loss for a 3-token sequence [2, 5, 3] if the model outputs logits as follows: P(2|1)=0.7, P(5|[1,2])=0.5, P(3|[1,2,5])=0.8

Loss = -[log(0.7) + log(0.5) + log(0.8)] = -[-0.357 + -0.693 + -0.223] = -[-1.273] = 1.273

Lower is better. If the model was perfect (all probabilities = 1.0), loss would be 0. Q4: Why is T5's span corruption objective more efficient than BERT's MLM?

BERT masks individual tokens (15% of tokens), treating each independently. T5 masks contiguous spans (average span length ~3 tokens, masking ~15% of text).

Span corruption is more efficient because:

1. Each span requires one prediction instead of multiple token predictions
2. The model learns to reconstruct coherent phrases, not isolated tokens
3. Fewer masked positions for the same amount of corruption

T5's approach is particularly beneficial for generation tasks where output is naturally structured in spans. Q5: GPT-3 has 175B parameters and was trained on 570GB of text. If each token is processed once per epoch, and 10 epochs were used, how many forward passes were performed?

Assuming ~0.75 bytes per token on average:

• 570GB = 570 × 10⁹ bytes
• Tokens ≈ 570 × 10⁹ / 0.75 ≈ 7.6 × 10¹¹ tokens
• 10 epochs = 7.6 × 10¹² tokens processed
• Each token requires one forward pass in CLM
• Total forward passes: ~7.6 trillion

This gives a sense of the scale of LLM pre-training. Q6: RoBERTa showed that removing NSP improves performance. Why might NSP be unnecessary?

RoBERTa found that NSP doesn't add value when training with longer sequences and more data. Possible reasons:

1. NSP is too easy — the model can exploit shallow features (topic shift) to solve it
2. The single-sentence training in BERT's NSP setup reduces effective sequence length
3. MLM alone with longer sequences captures sufficient sentence-level understanding

RoBERTa uses full-length sequences packed from multiple documents without the NSP task. Q7: Design a pre-training objective for a multimodal model (text + images).

Possibilities:

1. Masked modality modeling: Mask image patches or text tokens, predict from the other modality
2. Image-text matching: Predict if an image and caption match (like NSP but cross-modal)
3. Contrastive learning: Maximize similarity between matched image-text pairs, minimize for mismatched pairs
4. Prefix generation: Use image as prefix, generate text description

These approaches are used in models like CLIP, ALIGN, and Flamingo. Q8: Calculate the perplexity of a language model that assigns probability 0.25 to each token in a 10-token sequence.

Perplexity (PPL) = exp(Loss) Loss = -(10 × log(0.25)) = -(10 × -1.386) = 13.86 PPL = exp(13.86 / 10) = exp(1.386) = 4.0

Alternatively: PPL = 1 / average probability = 1 / 0.25 = 4.0

A perplexity of 4 means the model is as confused as if it had to choose uniformly among 4 tokens. Q9: What happens if you train BERT with 50% masking instead of 15%?

With 50% masking:

1. Too little context: The model has insufficient information to predict masked tokens
2. Unnatural task: Real text doesn't have half the words missing
3. Poor representations: The model doesn't learn good representations for non-masked tokens
4. Higher variance: Fewer training examples per sequence
5. Performance drops: Empirical studies show 15% is optimal for BERT-style models

Higher masking rates work better for T5's span corruption because contiguous spans provide more context for reconstruction. Q10: How does ELECTRA's Replaced Token Detection (RTD) achieve better sample efficiency than MLM?

ELECTRA uses a two-model approach:

1. Generator (small MLM): Predicts masked tokens
2. Discriminator (main model): Predicts whether each token is original or replaced

RTD is more efficient because:

• All tokens contribute to the loss (discriminator evaluates every position)
• The task (detect replacement) is harder than MLM prediction
• The discriminator learns from all positions, not just 15%
• ELECTRA achieves BERT-level performance with 1/4 the compute

Loss is computed over all N tokens, not just 0.15N masked ones.

7. 🔗 Cross-References

• Next: GPT Architecture (Week 4)
• Next: BERT Architecture (Week 5)
• Video: BSDA5004 Week 3-4 transcripts Join Discord PreviousLayer Norm & ResidualNextGPT Architecture