NLP Evaluation: BLEU, ROUGE, Perplexity

🎯 Learning Objectives

• Compute BLEU score with n-gram precision and brevity penalty
• Understand ROUGE's recall-based approach for summarization
• Calculate perplexity and relate it to model uncertainty
• Identify limitations of automatic evaluation metrics

📋 Prerequisites

• n-gram language models
• Precision and recall
• Machine translation concepts

1. 📖 Core Content

1.1 Why Automatic Evaluation?

1.2 BLEU Score (Bilingual Evaluation Understudy)

• $p_n$: Precision for n-grams (count of matching n-grams / total n-grams)
• $BP$: Brevity penalty (penalizes overly short translations)
• $w_n$: Weight for each n-gram length (typically $1/N$) 1 & \text{if } c > r \\ e^{(1-r/c)} & \text{if } c \leq r \end{cases}$$ Where c is candidate length, r is reference length. ```python # runnable from collections import Counter def bleu(candidate, reference, max_n=4): """ Compute BLEU score Args: candidate: Generated text (list of tokens) reference: Reference text (list of tokens) max_n: Maximum n-gram order """ c = len(candidate) r = len(reference) # Brevity penalty bp = 1.0 if c > r else np.exp(1 - r / c) # Precision for each n-gram log_precisions = [] for n in range(1, max_n + 1): cand_ngrams = Counter(tuple(candidate[i:i+n]) for i in range(len(candidate) - n + 1)) ref_ngrams = Counter(tuple(reference[i:i+n]) for i in range(len(reference) - n + 1)) # Count clipped matches matches = sum(min(cand_ngrams[ng], ref_ngrams.get(ng, 0)) for ng in cand_ngrams) total = max(len(candidate) - n + 1, 1) precision = matches / total if precision > 0: log_precisions.append(np.log(precision)) if not log_precisions: return 0.0 bleu = bp * np.exp(np.mean(log_precisions)) return bleu # Example import numpy as np cand = "the cat sat on the mat".split() ref = "the cat is on the mat".split() print(f"Candidate: {cand}") print(f"Reference: {ref}") print(f"BLEU score: {bleu(cand, ref):.4f}") ``` ### 1.3 ROUGE (Recall-Oriented Understudy for Gisting Evaluation) | Metric | Measures | Task | |--------|----------|------| | ROUGE-N | n-gram recall | Summarization | | ROUGE-L | Longest Common Subsequence | Captures fluency | | ROUGE-W | Weighted LCS | Prioritizes consecutive matches | | ROUGE-S | Skip-bigram | Allows gaps in matching | ### 1.4 Perplexity PPL = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log P(w_i | w_{<i})\right) Lower perplexity = better model. Perplexity of k means the model is as "confused" as if it had to choose uniformly among k options. ### 1.5 Limitations | Metric | Limitation | |--------|------------| | BLEU | No semantic understanding, favors shorter, favors reference-like text | | ROUGE | Recall-only (misses off-topic additions), no semantics | | Perplexity | Correlates weakly with generation quality | | Accuracy | Misleading for imbalanced classes | --- ## 📝 Practice Questions > Q1 > > <strong>Q1</strong>: Candidate: "the the the the" (4 words). Reference: "the cat sat" (3 words). Compute BLEU-1. > > Candidate length c=4, Reference length r=3 > BP = e^(1-3/4) = e^0.25 = 0.779 > > Unigram precision: candidate has 4 unigrams, all "the". Reference has 3 unigrams: "the", "cat", "sat". Clipped count: min(4, 1) = 1. > > p_1 = 1/4 = 0.25 > > BLEU-1 = 0.779 × 0.25 = 0.195 > > Even though "the" is correct, the BLEU score is low due to: > 1. Brevity penalty (shorter reference) > 2. Low precision (75% of generated words are wrong) > Q2 > > <strong>Q2 > > <strong>Q2</strong>: A language model assigns probability 0.02 to each word in a 100-word sentence. What's the perplexity? > > Average log probability = (1/100) × 100 × log(0.02) = log(0.02) = -3.912 > > PPL = exp(3.912) = 50 > > The model has perplexity of 50, meaning it's as confused as if it had to choose uniformly among 50 tokens. This is quite high — a good language model would have PPL < 30. > Q3 > > <strong>Q3 > > <strong>Q3 > > <strong>Q3</strong>: Why does BLEU use a brevity penalty? > > Without the brevity penalty, a system could achieve high precision by generating very short output — only the words it's confident about. For example, generating just "the" has perfect unigram precision if "the" appears in the reference. > > The brevity penalty penalizes outputs shorter than the reference. A system that generates "the" (short) gets penalized; a system that generates the full correct translation with appropriate length is rewarded. > > This makes BLEU a balanced metric: it requires both precision (correct words) and adequate length (completeness). > Q4 > > <strong>Q4 > > <strong>Q4 > > <strong>Q4 > > <strong>Q4 > > <strong>Q4</strong>: A summarization system generates "The cat sat." but the reference is "The cat sat on the mat." Compute ROUGE-1 recall. > > System unigrams: {The, cat, sat} > Reference unigrams: {The, cat, sat, on, the, mat} > > Overlap: {The, cat, sat} (3 unigrams match) > > ROUGE-1 recall = 3 / 6 = 0.5 > > The system captures 50% of the reference content. ROUGE doesn't penalize missing words like "on" and "mat" except through lower recall. This shows why ROUGE is recall-oriented — it measures how much of the reference is captured. </details> --- ## 🔗 Cross-References - Next: [Contextual Embeddings](../week11/11-contextual-embeddings.md) - Video: BSDA5005 Week 10 transcripts Join Discord PreviousSeq2Seq & AttentionNextBSDA5005 — Natural Language Processing (NLP)