Part-of-Speech Tagging with HMM & Viterbi

🎯 Learning Objectives

• Explain the POS tagging task and tag sets (Penn Treebank)
• Model POS tagging as a Hidden Markov Model
• Implement the Viterbi algorithm for decoding
• Handle unknown words and tag transition probabilities

📋 Prerequisites

• Markov chains and probability
• Text preprocessing basics

---

1. 📖 Core Content

1.1 What is POS Tagging?

• The/DT cat/NN sat/VBD on/IN the/DT mat/NN

1.2 POS Tag Sets

1.3 Hidden Markov Model for POS Tagging

• States: POS tags (NN, VB, JJ, ...)
• Observations: Words
• Transition probabilities: $P(t_i | t_{i-1})$ — probability of tag given previous tag
• Emission probabilities: $P(w_i | t_i)$ — probability of word given tag

$$P(T|W) = \prod_{i=1}^n P(t_i | t_{i-1}) \cdot P(w_i | t_i)$$

1.4 The Viterbi Algorithm

$$v_t(j) = \max_{i} v_{t-1}(i) \cdot a_{ij} \cdot b_j(o_t)$$

• $v_t(j)$: Viterbi score for tag j at position t
• $a_{ij}$: Transition probability from tag i to tag j
• $b_j(o_t)$: Emission probability of word o_t from tag j

python
Run Inline
# runnable
import numpy as np
def viterbi(obs, states, start_prob, trans_prob, emit_prob):
    """
    Viterbi algorithm for POS tagging
    Args:
        obs: List of observed words (indices)
        states: List of POS tag indices
        start_prob: Initial tag probabilities
        trans_prob: Transition matrix (n_tags × n_tags)
        emit_prob: Emission matrix (n_tags × vocab)
    Returns:
        best_path: Most likely tag sequence
    """
    n_states = len(states)
    n_obs = len(obs)
    # Initialize Viterbi and backpointer tables
    viterbi = np.zeros((n_states, n_obs))
    backpointer = np.zeros((n_states, n_obs), dtype=int)
    # Initialization step
    for s in range(n_states):
        viterbi[s, 0] = np.log(start_prob[s]) + np.log(emit_prob[s, obs[0]])
        backpointer[s, 0] = 0
    # Recursion step
    for t in range(1, n_obs):
        for s in range(n_states):
            # Find best previous state
            scores = viterbi[:, t-1] + np.log(trans_prob[:, s]) + np.log(emit_prob[s, obs[t]])
            viterbi[s, t] = np.max(scores)
            backpointer[s, t] = np.argmax(scores)
    # Termination step
    best_last = np.argmax(viterbi[:, -1])
    # Backtrack
    best_path = [best_last]
    for t in range(n_obs - 1, 0, -1):
        best_path.insert(0, backpointer[best_path[0], t])
    return best_path, viterbi
# Example: tagging "the cat sat"
obs = [0, 1, 2]  # word indices
states = [0, 1, 2]  # DT, NN, VB (simplified)
# Probabilities (simplified)
start_prob = np.array([0.5, 0.3, 0.2])  # P(DT), P(NN), P(VB) as first tag
trans_prob = np.array([
    [0.1, 0.7, 0.2],  # DT → ?
    [0.3, 0.4, 0.3],  # NN → ?
    [0.4, 0.3, 0.3]   # VB → ?
])
emit_prob = np.array([
    [0.8, 0.1, 0.1],  # DT → the, cat, sat
    [0.1, 0.7, 0.2],  # NN → the, cat, sat
    [0.1, 0.2, 0.7]   # VB → the, cat, sat
])
path, scores = viterbi(obs, states, start_prob, trans_prob, emit_prob)
tag_names = ['DT', 'NN', 'VB']
print(f"Words: the(0), cat(1), sat(2)")
print(f"Best tag sequence: {[tag_names[p] for p in path]}")

1.5 Handling Unknown Words

1. Morphological cues: Words ending in "-ing" → likely VBG (gerund)
2. Capitalization: Capitalized words → likely NNP (proper noun)
3. Suffix analysis: "-tion" → NN, "-ly" → RB, "-ed" → VBD
4. Default distribution: Uniform or based on tag frequency

---

📝 Practice Questions

Q1: Compute P(DT → NN → VB) for the sequence "the dog runs" given transition probabilities P(DT|START)=0.5, P(NN|DT)=0.7, P(VB|NN)=0.3

P(DT, NN, VB | the, dog, runs) = P(DT|START) × P(the|DT) × P(NN|DT) × P(dog|NN) × P(VB|NN) × P(runs|VB)

Assuming emission probabilities: P(the|DT)=0.8, P(dog|NN)=0.7, P(runs|VB)=0.7

Probability = 0.5 × 0.8 × 0.7 × 0.7 × 0.3 × 0.7 = 0.04116 Q2

<strong>Q2

<strong>Q2</strong>: Why might HMM-based POS tagging perform poorly on rare words?

HMMs rely on emission probabilities P(word|tag). For rare words that didn't appear in training, emission probability is 0 for all tags (or smoothed to a very small value). The Viterbi algorithm then can't distinguish between possible tags based on word evidence, forcing it to rely entirely on transition probabilities.

Example: The rare word "xylophone" has never been seen in training. The model must guess its tag based only on surrounding tag patterns. If the previous word is "the" (DT), both NN and JJ are possible following DT.

Solutions: Use morphological clues (suffix analysis), better smoothing, or subword information. Q3

<strong>Q3

<strong>Q3</strong>: Compare HMM-based tagging with neural (BiLSTM/Transformer) tagging.

Neural models excel because they can learn complex patterns: character-level features handle unknown words, bidirectional context captures long-range dependencies, and attention mechanisms weight context appropriately. Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Given emission probabilities P(word|tag) and transition probabilities P(tag|prev_tag), derive the formula for the most likely tag sequence.

We want: $\arg\max_T P(T|W)$

By Bayes' rule: $P(T|W) = \frac{P(W|T) \cdot P(T)}{P(W)}$

Since $P(W)$ is constant for the argmax: $\arg\max_T P(T|W) = \arg\max_T P(W|T) \cdot P(T)$

Using the Markov assumption (tag only depends on previous tag) and independence assumption (word only depends on its tag): $P(W|T) \cdot P(T) = \prod_{i=1}^n P(w_i|t_i) \cdot P(t_1) \cdot \prod_{i=2}^n P(t_i|t_{i-1})$

Taking log (for numerical stability): $\arg\max_T \sum_{i=1}^n [\log P(w_i|t_i) + \log P(t_i|t_{i-1})]$

This is what Viterbi computes efficiently using dynamic programming.

</details> * * * ## 🔗 Cross-References - **Next**: [Parsing & Syntax](/courses/bsda5005/notes/.%2Fweek03%2F03-parsing-syntax) - **Video**: BSDA5005 Week 2 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**NLP Intro & Preprocessing**](/notes/04-degree-electives-bsda5005-nlp-week01-01-nlp-intro-preprocessing)[Next**Named Entity Recognition**](/notes/04-degree-electives-bsda5005-nlp-week04-04-named-entity-recognition)