NLP Introduction & Text Preprocessing

🎯 Learning Objectives

• Understand the levels of linguistic analysis in NLP
• Apply text preprocessing techniques: tokenization, normalization, stemming, lemmatization
• Explain the challenges of natural language ambiguity
• Implement a basic text preprocessing pipeline

📋 Prerequisites

• Basic Python programming
• Understanding of regular expressions

1. 📖 Core Content

1.1 What is NLP?

1.2 Levels of Linguistic Analysis

1.3 The Ambiguity Challenge

1.4 Text Preprocessing Pipeline

python
Run Inline
# runnable
import re
from collections import Counter
class TextPreprocessor:
    """Basic text preprocessing pipeline"""
    def __init__(self):
        self.word_counts = Counter()
    def tokenize(self, text):
        """Tokenize text into words"""
        # Split on whitespace and punctuation
        tokens = re.findall(r'\b\w+\b', text.lower())
        return tokens
    def normalize(self, token):
        """Basic normalization"""
        return token.lower()
    def remove_stopwords(self, tokens, stopwords=None):
        """Remove common stop words"""
        if stopwords is None:
            stopwords = {'the', 'a', 'an', 'is', 'was', 'were',
                        'in', 'on', 'at', 'to', 'for', 'of',
                        'and', 'or', 'but', 'it', 'its'}
        return [t for t in tokens if t not in stopwords]
    def stem(self, word):
        """Simple stemmer (Porter-like rules)"""
        # This is a simplified stemmer for demonstration
        if word.endswith('ing'):
            return word[:-3]
        if word.endswith('ed'):
            return word[:-2]
        if word.endswith('ly'):
            return word[:-2]
        if word.endswith('es'):
            return word[:-2]
        if word.endswith('s') and not word.endswith('ss'):
            return word[:-1]
        return word
    def process(self, text):
        """Full preprocessing pipeline"""
        tokens = self.tokenize(text)
        tokens = [self.normalize(t) for t in tokens]
        tokens = self.remove_stopwords(tokens)
        tokens = [self.stem(t) for t in tokens]
        return tokens
# Example
preprocessor = TextPreprocessor()
text = "The cats were running quickly through the city streets"
processed = preprocessor.process(text)
print(f"Original: {text}")
print(f"Processed: {processed}")

1.5 Stemming vs Lemmatization

📝 Practice Questions

Q1: Tokenize: "Dr. Smith's cat (the one with the hat!) ran away."

Simple whitespace+punctuation tokenization: ["Dr", "Smith", "s", "cat", "the", "one", "with", "the", "hat", "ran", "away"]

Note challenges:

• "Dr." contains a period (not end of sentence)
• "Smith's" could be ["Smith", "'s"] or ["Smiths"]
• Parentheses and exclamation mark are punctuation

Better tokenization: ["Dr.", "Smith's", "cat", "(", "the", "one", "with", "the", "hat", "!", ")", "ran", "away", "."] Q2: What are the three types of ambiguity in "Flying planes can be dangerous"?

1. Lexical ambiguity: "Flying" can be an adjective (planes that fly) or a verb (the act of flying)
2. Syntactic ambiguity:
   • [Flying planes] can be dangerous (planes that fly are dangerous)
   • Flying [planes can be dangerous] (the act of flying planes is dangerous)
3. Semantic ambiguity: "Dangerous" — dangerous to whom? The pilot? People on the ground?

This classic example shows how a simple 5-word sentence has multiple valid interpretations. Q3: Compare character n-grams, word n-grams, and subword tokenization for the sentence "I love NLP."

Character bigrams (n=2): ["I ", "lo", "ov", "ve", "e ", "N", "NL", "LP", "P."] Word bigrams (n=2): ["I love", "love NLP", "NLP."] Subword (BPE): Could split "NLP" if rare, or keep as one token

Character: Most granular, works for any language, OOV impossible Word: Semantically meaningful, OOV issues, large vocabulary Subword: Best balance, handles OOV, moderate vocabulary Q4: Why is "I ate the pizza with pepperoni" unambiguous while "I ate the pizza with a fork" uses the same structure but different meaning?

Both have the same syntactic structure: "I ate [the pizza] [with pepperoni/fork]"

• "with pepperoni" → modifier of pizza (what kind of pizza)
• "with a fork" → instrument of eating (how I ate)

The resolution requires world knowledge: we know pepperoni is a pizza topping and a fork is an eating utensil. The syntax is ambiguous, but semantics and pragmatics resolve it.

This is PP-attachment ambiguity — the prepositional phrase "with X" could attach to the noun phrase ("pizza") or the verb phrase ("ate"). Humans resolve it effortlessly using semantics.

🔗 Cross-References

• Next: POS Tagging
• Video: BSDA5005 Week 1 transcripts Join Discord NextPOS Tagging