Tokenization: BPE, WordPiece, SentencePiece

🎯 Learning Objectives

• Explain why subword tokenization is necessary for LLMs
• Implement Byte Pair Encoding (BPE)
• Understand WordPiece and SentencePiece algorithms
• Compare different tokenization approaches and their trade-offs

📋 Prerequisites

• Basic NLP preprocessing concepts
• Character encoding (UTF-8)

1. 📖 Core Content

1.1 The Tokenization Problem

1.2 Byte Pair Encoding (BPE)

Intuition

1. Start with characters as tokens
2. Count all adjacent pairs
3. Merge the most frequent pair
4. Repeat until desired vocabulary size

Worked Example

• "lo": 8 occurrences
• "ow": 8 occurrences
• "we": 3 occurrences
• "er": 2 occurrences
• "es": 1 occurrence
• "st": 1 occurrence Merge "lo" → "lo" Corpus: lo w _ lo w e r _ lo w e s t _ lo w _ lo w _ lo w e r Vocabulary: {l, o, w, e, r, s, t, lo} Step 2: Count pairs
• "ow": 8 occurrences
• "lo" + "w" → "low": 8 occurrences
• "we": 3 occurrences
• "er": 2 occurrences Merge "ow" → "ow" Corpus: lo ow _ lo ow e r _ lo ow e s t _ lo ow _ lo ow _ lo ow e r Better: Merge "lo"+"w" → "low" 🠖 actually let's merge "low": After merging "lo", the most frequent pair is "ow" (8). But "low" appears 8 times too. Let's say we merge "low": Corpus: low _ low e r _ low e s t _ low _ low _ low e r Vocabulary: {l, o, w, e, r, s, t, low} Step 3: Count pairs
• "l"+"o" doesn't exist anymore (it's "low")
• "we": 3 occurrences ("low"+"e", but "e" is a separate token)
• Actually "low" + " " + "low" → "low" is now a single token Let me simplify with a clearer example.

python
Run Inline
# runnable
from collections import Counter
import re
def bpe_train(corpus, vocab_size):
    """
    Train BPE tokenizer
    Args:
        corpus: List of strings
        vocab_size: Target vocabulary size
    Returns:
        vocab: Set of tokens
        merges: List of merges performed
    """
    # Initialize vocabulary with characters
    vocab = set()
    for text in corpus:
        vocab.update(list(text))
    # Add special tokens
    special_tokens = ['<unk>', '<s>', '</s>']
    vocab.update(special_tokens)
    # Split corpus into characters with word boundaries
    words = [list(word) + ['</w>'] for word in corpus[0].split()]
    merges = []
    while len(vocab) < vocab_size:
        # Count adjacent pairs
        pairs = Counter()
        for word in words:
            for i in range(len(word) - 1):
                pairs[(word[i], word[i+1])] += 1
        if not pairs:
            break
        # Find most frequent pair
        best_pair = pairs.most_common(1)[0][0]
        merges.append(best_pair)
        # Merge the pair
        new_words = []
        for word in words:
            new_word = []
            i = 0
            while i < len(word):
                if i < len(word) - 1 and (word[i], word[i+1]) == best_pair:
                    new_word.append(word[i] + word[i+1])
                    i += 2
                else:
                    new_word.append(word[i])
                    i += 1
            new_words.append(new_word)
        words = new_words
        # Add new token to vocab
        vocab.add(best_pair[0] + best_pair[1])
    return vocab, merges
corpus = ["low lowest lower low low lower"]
vocab, merges = bpe_train(corpus, 20)
print(f"Vocabulary: {sorted(vocab)}")
print(f"Merges: {merges}")

1.3 BPE in GPT Models

• Vocabulary size: 50,257
• Base tokens are bytes (256 values), not characters
• Can encode any Unicode string
• No [UNK] token needed (everything is representable)

1.4 WordPiece

Used in: BERT, DistilBERT

1.5 SentencePiece

Used in: T5, LLaMA, ALBERT

1. No language-specific preprocessing: Doesn't rely on spaces (works for Chinese, Japanese)
2. Two training algorithms: Can use BPE or Unigram LM
3. Directly models spaces: Space is a regular token

Unigram LM Tokenization

1. Start with a large vocabulary (all possible substrings up to some length)
2. Compute the likelihood of the corpus under a unigram model
3. Iteratively remove tokens that least reduce likelihood
4. Stop at target vocabulary size Loss function:

1.6 Tokenization Comparison

1.7 Why This Matters

• Vocabulary coverage: Larger vocab = fewer tokens per word = efficient processing
• OOV handling: Subword tokenization eliminates unknown words
• Language bias: Space-based tokenizers underperform on Chinese/Japanese
• Token efficiency: Some languages need 2× more tokens for the same meaning
• Adversarial robustness: Token boundaries affect which inputs are "adversarial"

6. 📝 Practice Questions

Q1: For the corpus "aa ab aa ab ba", show the first two BPE merges.

Initial tokens: [a, a, _, a, b, _, a, a, _, a, b, _, b, a]

Step 1: Count pairs "aa": 2 (positions 0-1, 6-7) "a_": 2 (positions 1-2, 7-8) "a": 2 (positions 2-3, 8-9) "ab": 2 (positions 3-4, 9-10) "b": 2 (positions 4-5, 10-11) "_b": 1 "ba": 1

Ties broken arbitrarily. Let's merge "aa" first: Corpus: aa _ ab _ aa _ ab _ ba

Step 2: Count pairs again "aa_": 2 "a": 2 "ab": 2 "b": 2 "_b": 1 "ba": 1

Merge "ab" (or any other pair with count 2): Corpus: aa _ ab _ aa _ ab _ ba Q2

<strong>Q2</strong>: Why does SentencePiece not need pre-tokenization?

SentencePiece treats the raw input as a sequence of bytes or Unicode characters. It doesn't assume word boundaries (spaces). Instead, it includes the space character as a regular token (often denoted as "_" or "▁"). This means the tokenizer can learn subword units across word boundaries, and it works for languages like Chinese, Japanese, or Thai that don't use spaces.

During decoding, spaces are simply concatenated like any other token. Q3: GPT-2 has 50,257 vocabulary. If each token embedding is d_model=768, what is the size of the embedding matrix?

Embedding matrix: 50,257 × 768 = 38,597,376 parameters ≈ 38.6M

This is the embedding layer (token embeddings). GPT-2 also has positional embeddings (1024 × 768 = 0.79M). Combined, the embedding layers have ~39.4M parameters out of GPT-2 Small's total 124M (~32% of all parameters).

Large vocabularies significantly increase model size through the embedding layer. Q4: A word "internationalization" takes 20 characters. With BPE tokens ["inter", "national", "ization"], how many tokens does it become?

It becomes 3 tokens: ["inter", "national", "ization"] instead of 20 characters or a single [UNK] token.

This is ~7× more efficient than character-level (3 vs 20) and avoids the OOV problem of word-level tokenization. Q5: What happens if BPE vocabulary is too small (e.g., 1000 tokens)?

With 1000 tokens:

1. Most words are split into many subword pieces ("unbelievable" → ["un", "bel", "iev", "ab", "le"])
2. Sequence lengths increase → computational cost increases
3. Each subword carries less semantic meaning
4. The model must learn compositionality of subword pieces
5. Potential for "tokenization artifacts" where unfortunate splits create confusing pieces

Typical BERT-style vocabularies are 30K-50K, balancing coverage and efficiency. Q6

<strong>Q6</strong>: How does BPE handle the word "don't" when apostrophes are rare in the training corpus?

"don't" might be tokenized as:

• ["don", "'", "t"] if apostrophe is infrequent
• ["do", "n't"] if "n't" is a common merge
• ["don't"] if the word appears frequently enough

The exact tokenization depends on the frequency of "don't" and its components in the training corpus. Common contractions like "n't" often become their own tokens because they appear frequently. Q7

<strong>Q7</strong>: Why might a model tokenize "hello" and "Hello" differently?

Case-sensitive tokenizers treat "h" and "H" as different characters. If the corpus has more lowercase "hello" than capitalized "Hello", the tokenizer might:

• Merge "he" + "llo" → "hello" as a single token (frequent)
• Keep "H" + "ello" as separate tokens (less frequent)

This means "hello" is 1 token and "Hello" is 2 tokens, which affects the model's understanding and generation. Some models use lowercasing or case-preserving tokenization to handle this. Q8

<strong>Q8</strong>: In SentencePiece Unigram, how is the optimal segmentation of a sequence determined?

SentencePiece uses the Viterbi algorithm to find the most likely segmentation. Given the unigram probabilities of each subword token, Viterbi finds:

$\text{segmentation}(X) = \arg\max_{x_1,...,x_k} \prod_{i=1}^k P(x_i)$

This is a classic dynamic programming problem — the probability of segmenting up to position i is:

$P(i) = \max_{j < i} P(j) \cdot P(\text{token}(j,i))$

Viterbi efficiently finds the optimal segmentation in O(n × max_token_length). Q9

<strong>Q9</strong>: A BPE tokenizer trained primarily on English text is used for German. What issues might arise?

Potential issues:

1. Compound words: German has long compounds ("Donaudampfschifffahrtsgesellschaftskapitän") that might be split into many pieces
2. Different character distributions: German has "ä", "ö", "ü", "ß" which may be rare in English training data, causing splits
3. Morphology differences: German has more inflectional morphology, requiring more subwords per word
4. Efficiency drop: German text might need 30-50% more tokens than English for the same meaning
5. Suboptimal merges: Merges that are useful for English may not be optimal for German

Solutions: Joint training on multilingual data, or language-specific tokenizers. Q10

<strong>Q10</strong>: BPE vocabulary is typically 50K-100K tokens. If we double it to 200K, what trade-offs occur?

Pros:

• Fewer tokens per sequence (common words remain whole)
• Lower effective sequence length → faster processing
• Less subword composition needed
• Better coverage of rare words

Cons:

• Larger embedding matrix → more parameters (200K × 768 = 154M params)
• Less parameter sharing across related words ("run" and "running" might be separate)
• Potential overfitting to training corpus vocabulary
• Slower final linear layer (vocabulary × d_model)
• Memory: logits computation becomes more expensive

The optimal vocabulary size balances token efficiency against model capacity.

7. 🔗 Cross-References

• Previous: BERT Architecture
• Next: Fine-tuning Methods (Week 7)
• Video: BSDA5004 Week 6 transcripts Join Discord PreviousBERT ArchitectureNextFine-tuning & PEFT