Neural Sync Active

Text Classification with TF-IDF and Deep Learning

Registry Synced

Text Classification with TF-IDF and Deep Learning

822 words

4 min read

Reading compass

Now · 🎯 Learning Objectives

Text Classification with TF-IDF and Deep Learning

🎯 Learning Objectives

Extract TF-IDF features from text
Implement Naive Bayes and logistic regression for text classification
Understand the limitations of bag-of-words representations
Build a text classifier with neural networks

📋 Prerequisites

Word embeddings or bag-of-words representation
Basic classification algorithms

1. 📖 Core Content

1.1 The Text Classification Problem

Task: Assign a category to a piece of text. Examples: Spam detection, sentiment analysis, topic labeling, intent detection. Challenges: Variable-length input, sparse features, context dependence.

1.2 TF-IDF Feature Extraction

TF-IDF = Term Frequency × Inverse Document Frequency

TF(t, d) = \frac{\text{count}(t, d)}{\sum_{t' \in d} \text{count}(t', d)}

IDF(t) = \log\frac{N}{1 + DF(t)}

TF\text{-}IDF(t, d) = TF(t, d) \times IDF(t)

Where:

$N$ : Total number of documents
$DF(t)$ : Number of documents containing term t


python
# runnable
import numpy as np
from collections import Counter
def compute_tfidf(documents):
    """Compute TF-IDF for a set of documents"""
    N = len(documents)
    df = Counter()  # Document frequency
    all_terms = set()
    # Tokenize and count
    doc_terms = []
    for doc in documents:
        terms = doc.lower().split()
        doc_terms.append(terms)
        df.update(set(terms))
        all_terms.update(terms)
    # Compute TF-IDF for each document
    vocab = sorted(all_terms)
    tfidf_matrix = np.zeros((N, len(vocab)))
    for i, terms in enumerate(doc_terms):
        tf = Counter(terms)
        max_tf = max(tf.values()) if tf else 1
        for j, term in enumerate(vocab):
            tf_val = tf.get(term, 0) / max_tf
            idf_val = np.log(N / (1 + df.get(term, 0)))
            tfidf_matrix[i, j] = tf_val * idf_val
    return tfidf_matrix, vocab
# Example
docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are pets",
    "the mat was under the cat"
]
matrix, vocab = compute_tfidf(docs)
print(f"TF-IDF matrix shape: {matrix.shape}")
print(f"Vocabulary: {vocab}")
print(f"\nDocument 0 TF-IDF vector:\n{np.round(matrix[0], 3)}")

1.3 Classification Algorithms

Algorithm	Pros	Cons	Best For
Naive Bayes	Fast, small data	Feature independence assumption	Baseline, small data
Logistic Regression	Interpretable, calibrated	Linear decision boundary	Binary classification
SVM	Works well with high-dim data	Slow to train	Text categorization
Neural Networks	Captures complex patterns	Needs lots of data	Large-scale classification

📝 Practice Questions

Q1

Q1: For a corpus of 1000 documents, the word "algorithm" appears in 50 documents. A specific document has "algorithm" appearing 3 times out of 100 total words. Compute TF-IDF for "algorithm" in this document.

TF = 3/100 = 0.03 IDF = log(1000 / (1 + 50)) = log(1000/51) = log(19.61) = 2.98

TF-IDF = 0.03 × 2.98 = 0.089

The word "algorithm" has moderate importance in this document — it's relatively rare across the corpus (IDF boost) but appears frequently in this document. Q2

Q2

Q2: Why is IDF important? What happens if a common word (e.g., "the") appears in every document?

For "the" appearing in all 1000 documents: IDF = log(1000 / (1 + 1000)) = log(1000/1001) = log(0.999) ≈ -0.001

IDF ≈ 0, so TF-IDF ≈ 0 for "the" in all documents. This is correct — "the" carries no discriminative information for classification.

Without IDF, TF would give high weight to common words, making "the" seem important. IDF suppresses words that appear uniformly across the corpus — they're NOT discriminative. Q3

Q3

Q3

Q3

Q3: Why does Naive Bayes work well for text despite the independence assumption being clearly violated?

Naive Bayes assumes features (words) are independent given the class — clearly false ("sky" and "blue" co-occur). Yet it works because:

Decision boundaries: Even with poor probability estimates, the decision boundary can be correct

Feature sparsity: Most word pairs don't co-occur, so independence violations are limited

Relative comparison: Classification depends on which class has higher probability, not absolute values

Feature selection: Using only top features reduces independence violations

In text, Naive Bayes is often competitive with more sophisticated methods, especially on small datasets. Q4

Q4

Q4

Q4

Q4

Q4: A binary sentiment classifier achieves 99% accuracy on a test set where 98% of reviews are positive. Is this good?

No! The baseline accuracy (always predicting the majority class) is 98%. The model's 99% is only 1% better than a trivial baseline.

This is a class imbalance problem. Better metrics:

Precision: Of predicted positive reviews, how many are correct?

Recall: Of actual positive reviews, how many were found?

F1-Score: Harmonic mean of precision and recall

AUC-ROC: Measures ranking quality independent of threshold

The confusion matrix would reveal the model's actual performance: if it misses most negative reviews (predicting "positive" for everything), accuracy is high but the model is useless.

</details> * * * ## 🔗 Cross-References - **Next**: [RNNs & LSTMs](../week07/07-rnn-lstm-nlp.md) - **Video**: BSDA5005 Week 6 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Word Embeddings**](/notes/04-degree-electives-bsda5005-nlp-week05-05-word-embeddings)[Next**Seq2Seq & Attention**](/notes/04-degree-electives-bsda5005-nlp-week08-08-seq2seq-attention)