BERT Architecture: Encoder-Only & Bidirectional Context

🎯 Learning Objectives

• Understand BERT's encoder-only architecture and bidirectional attention
• Explain how MLM + NSP pre-training works
• Adapt BERT to downstream tasks via fine-tuning
• Compare BERT with GPT on architecture and use cases

📋 Prerequisites

• Pre-training objectives (MLM, NSP) from Week 3
• Transformer encoder architecture
• Fine-tuning concepts

1. 📖 Core Content

1.1 BERT Overview

• Bidirectional self-attention (no causal mask)
• Learned positional embeddings
• [CLS] token for classification
• [SEP] token for separating sentences (Diagram)

1.2 BERT Input Representation

1.3 Pre-training: MLM + NSP

1. Masked Language Model (MLM): 15% tokens masked, predict original
2. Next Sentence Prediction (NSP): Predict if sentence B follows sentence A Training data: BooksCorpus (800M words) + English Wikipedia (2,500M words) Training hyperparameters:

• BERT-Base: L=12, H=768, A=12, 110M params
• BERT-Large: L=24, H=1024, A=16, 340M params
• Batch size: 256 sequences
• Learning rate: 1e-4
• Training steps: 1,000,000
• Adam optimizer with weight decay

1.4 Fine-tuning BERT

Classification

Named Entity Recognition (NER)

Question Answering

1.5 BERT Variants

1.6 BERT vs GPT

1.7 Why This Matters

• Text classification (sentiment, spam, topic)
• Named Entity Recognition
• Question Answering
• Sentence similarity / paraphrase detection Most modern "embedding models" (text-embedding-3-small, etc.) are based on the BERT architecture.

6. 📝 Practice Questions

Q1: If BERT-Base has d_model=768, A=12 heads, what is d_k per head?

d_k = d_model / A = 768 / 12 = 64

Each head in BERT-Base operates in 64-dimensional space. Q2: Why does BERT use [CLS] token for classification?

The [CLS] token is designed to aggregate information from the entire sequence. Because BERT's attention is bidirectional, [CLS] attends to all tokens and all tokens attend to [CLS]. The final [CLS] representation is a contextualized summary of the entire input, making it suitable for classification.

In contrast, GPT uses the last token (which has attended to all previous tokens) for classification. Q3: How does RoBERTa differ from BERT in pre-training?

RoBERTa (Robustly Optimized BERT Approach) changes:

1. Removes NSP objective
2. Dynamic masking (different mask pattern per epoch vs BERT's static)
3. Larger batches (8K vs 256)
4. More data (160GB vs 16GB)
5. Longer training (500K steps with larger batches)
6. Higher learning rate

These changes make RoBERTa outperform BERT with the same architecture. Q4: For question answering with BERT, why are start and end logits computed separately?

The answer span is defined by a start index i and end index j (i ≤ j). Instead of predicting all possible (i,j) pairs (O(n²)), BERT decomposes the problem:

1. Compute P(start = i) for each position i
2. Compute P(end = j) for each position j
3. The best span is: argmax_{i ≤ j} P(start=i) × P(end=j)

This is O(n) per position and O(n²) for finding the best pair, which is tractable. Q5: BERT-Large has 340M parameters. If fine-tuning adds 10K parameters for a classification head, what fraction are task-specific?

Fraction = 10,000 / 340,000,000 = 0.0000294 = 0.00294%

Only ~0.003% of parameters are task-specific. The remaining 99.997% are shared from pre-training. This is why fine-tuning is so effective — we leverage the vast majority of parameters that already encode general language understanding. Q6: Why can't BERT be used for text generation?

BERT is an encoder-only model with bidirectional attention. Generation requires:

1. Autoregressive left-to-right processing (BERT processes all tokens at once)
2. Causal masking (BERT has no masking)
3. A way to produce the next token (BERT doesn't predict the next token)

To generate with BERT, you'd need to add a decoder on top (making it encoder-decoder) or use iterative masking/unmasking (slow and impractical). Q7: What is the WordPiece tokenization vocabulary size of BERT-Base?

BERT-Base uses WordPiece with a vocabulary of 30,000 tokens. This was chosen to balance coverage (most words are in the vocabulary) with efficiency (rare words are split into subwords).

Compare to:

• GPT-2: 50,257 BPE tokens
• LLaMA: 32,000 SentencePiece tokens
• T5: 32,000 SentencePiece tokens Q8: BERT processes sequences up to 512 tokens. If a document has 2000 tokens, how can BERT handle it?

Options:

1. Sliding window: Process windows of 512 tokens with overlap, aggregate results
2. Truncation: Keep first 512 tokens (loses later information)
3. Hierarchical: Split into chunks, encode each, then combine representations
4. Longformer/XLNet: Use specialized architectures designed for longer sequences

BERT itself can't process >512 tokens due to positional embedding limit. Later models (Longformer, BigBird) address this with sparse attention. Q9: In NER fine-tuning with BERT, why might the model perform poorly on rare entity types?

For rare entity types (e.g., "LAW" or "PRODUCT" in a domain-specific dataset):

1. Limited labeled examples: The fine-tuning head sees few examples
2. Class imbalance: Rare types are dominated by "O" (non-entity) tags
3. Pre-training bias: BERT was pre-trained on general text, not domain-specific entities

Solutions:

• Use data augmentation for rare types
• Apply class weighting in the loss function
• Use few-shot or prompting approaches instead of fine-tuning
• Domain-adapt the pre-trained model on relevant text before fine-tuning Q10

<strong>Q10</strong>: How would you modify BERT for a span-based question answering task where answers can span multiple sentences?

Standard BERT QA extracts a contiguous span (start ≤ end) from a single context passage. For multi-sentence answers:

1. Longer context: Use a model that supports longer sequences (XLNet, Longformer)
2. Multiple spans: Add a "number of spans" classifier and extract multiple (start, end) pairs
3. Two-stage: First identify relevant sentence(s), then extract span within each
4. Sequence-to-sequence: Use T5 or BART which generate the answer token by token (allowing multi-sentence answers)

The last approach (seq2seq) is often preferred for multi-sentence outputs as it doesn't require the span constraint.

7. 🔗 Cross-References

• Previous: GPT Architecture
• Next: Tokenization (Week 6)
• Video: BSDA5004 Week 5 transcripts Join Discord PreviousGPT ArchitectureNextTokenization Methods