Text Classification with TF-IDF and Deep Learning

🎯 Learning Objectives

• Extract TF-IDF features from text
• Implement Naive Bayes and logistic regression for text classification
• Understand the limitations of bag-of-words representations
• Build a text classifier with neural networks

📋 Prerequisites

• Word embeddings or bag-of-words representation
• Basic classification algorithms

---

1. 📖 Core Content

1.1 The Text Classification Problem

1.2 TF-IDF Feature Extraction

$$TF(t, d) = \frac{\text{count}(t, d)}{\sum_{t' \in d} \text{count}(t', d)}$$ $$IDF(t) = \log\frac{N}{1 + DF(t)}$$ $$TF\text{-}IDF(t, d) = TF(t, d) \times IDF(t)$$

• $N$: Total number of documents
• $DF(t)$: Number of documents containing term t

python
Run Inline
# runnable
import numpy as np
from collections import Counter
def compute_tfidf(documents):
    """Compute TF-IDF for a set of documents"""
    N = len(documents)
    df = Counter()  # Document frequency
    all_terms = set()
    # Tokenize and count
    doc_terms = []
    for doc in documents:
        terms = doc.lower().split()
        doc_terms.append(terms)
        df.update(set(terms))
        all_terms.update(terms)
    # Compute TF-IDF for each document
    vocab = sorted(all_terms)
    tfidf_matrix = np.zeros((N, len(vocab)))
    for i, terms in enumerate(doc_terms):
        tf = Counter(terms)
        max_tf = max(tf.values()) if tf else 1
        for j, term in enumerate(vocab):
            tf_val = tf.get(term, 0) / max_tf
            idf_val = np.log(N / (1 + df.get(term, 0)))
            tfidf_matrix[i, j] = tf_val * idf_val
    return tfidf_matrix, vocab
# Example
docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are pets",
    "the mat was under the cat"
]
matrix, vocab = compute_tfidf(docs)
print(f"TF-IDF matrix shape: {matrix.shape}")
print(f"Vocabulary: {vocab}")
print(f"\nDocument 0 TF-IDF vector:\n{np.round(matrix[0], 3)}")

1.3 Classification Algorithms

---

📝 Practice Questions

Q1

<strong>Q1</strong>: For a corpus of 1000 documents, the word "algorithm" appears in 50 documents. A specific document has "algorithm" appearing 3 times out of 100 total words. Compute TF-IDF for "algorithm" in this document.

TF = 3/100 = 0.03 IDF = log(1000 / (1 + 50)) = log(1000/51) = log(19.61) = 2.98

TF-IDF = 0.03 × 2.98 = 0.089

The word "algorithm" has moderate importance in this document — it's relatively rare across the corpus (IDF boost) but appears frequently in this document. Q2

<strong>Q2

<strong>Q2</strong>: Why is IDF important? What happens if a common word (e.g., "the") appears in every document?

For "the" appearing in all 1000 documents: IDF = log(1000 / (1 + 1000)) = log(1000/1001) = log(0.999) ≈ -0.001

IDF ≈ 0, so TF-IDF ≈ 0 for "the" in all documents. This is correct — "the" carries no discriminative information for classification.

Without IDF, TF would give high weight to common words, making "the" seem important. IDF suppresses words that appear uniformly across the corpus — they're NOT discriminative. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: Why does Naive Bayes work well for text despite the independence assumption being clearly violated?

Naive Bayes assumes features (words) are independent given the class — clearly false ("sky" and "blue" co-occur). Yet it works because:

1. Decision boundaries: Even with poor probability estimates, the decision boundary can be correct
2. Feature sparsity: Most word pairs don't co-occur, so independence violations are limited
3. Relative comparison: Classification depends on which class has higher probability, not absolute values
4. Feature selection: Using only top features reduces independence violations

In text, Naive Bayes is often competitive with more sophisticated methods, especially on small datasets. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: A binary sentiment classifier achieves 99% accuracy on a test set where 98% of reviews are positive. Is this good?

No! The baseline accuracy (always predicting the majority class) is 98%. The model's 99% is only 1% better than a trivial baseline.

This is a class imbalance problem. Better metrics:

• Precision: Of predicted positive reviews, how many are correct?
• Recall: Of actual positive reviews, how many were found?
• F1-Score: Harmonic mean of precision and recall
• AUC-ROC: Measures ranking quality independent of threshold

The confusion matrix would reveal the model's actual performance: if it misses most negative reviews (predicting "positive" for everything), accuracy is high but the model is useless.

</details> * * * ## 🔗 Cross-References - **Next**: [RNNs & LSTMs](/courses/bsda5005/notes/.%2Fweek07%2F07-rnn-lstm-nlp) - **Video**: BSDA5005 Week 6 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Word Embeddings**](/notes/04-degree-electives-bsda5005-nlp-week05-05-word-embeddings)[Next**Seq2Seq & Attention**](/notes/04-degree-electives-bsda5005-nlp-week08-08-seq2seq-attention)