Naive Bayes Classifier

🎯 Learning Objectives

• Derive Naive Bayes from Bayes' theorem
• Explain the "naive" conditional independence assumption
• Implement Gaussian, Multinomial, and Bernoulli Naive Bayes
• Apply Naive Bayes to text classification

📖 Core Content

2.1 Intuition: The Simple-but-Surprisingly-Effective Classifier

2.2 Bayes' Theorem

• $P(y | x)$: posterior — probability of class y given features x
• $P(x | y)$: likelihood — probability of features x given class y
• $P(y)$: prior — probability of class y (before seeing data)
• $P(x)$: evidence — probability of features x (marginal)

2.3 The Naive Assumption

2.4 Types of Naive Bayes

2.5 Text Classification Example

• Spam: "free money", "win free money", "money money"
• Not Spam: "meeting lunch", "free lunch meeting", "meeting" Step 1: Compute priors. P(Spam) = 3/6 = 0.5, P(Not Spam) = 3/6 = 0.5. Step 2: Compute likelihoods with Laplace smoothing ($\alpha = 1$). $P(\text{"free"} | \text{Spam}) = \frac{2 + 1}{5 + 5} = \frac{3}{10} = 0.3$ (2 occurrences in Spam, 5 total words in Spam, 5 vocab size) $P(\text{"meeting"} | \text{Spam}) = \frac{0 + 1}{10} = 0.1$ $P(\text{"free"} | \text{Not Spam}) = \frac{1 + 1}{6 + 5} = \frac{2}{11} \approx 0.182$ $P(\text{"meeting"} | \text{Not Spam}) = \frac{3 + 1}{11} = \frac{4}{11} \approx 0.364$ Step 3: Classify "free meeting":
• $P(\text{Spam}) \times P(\text{free}|\text{Spam}) \times P(\text{meeting}|\text{Spam}) = 0.5 \times 0.3 \times 0.1 = 0.015$
• $P(\text{Not Spam}) \times P(\text{free}|\text{Not Spam}) \times P(\text{meeting}|\text{Not Spam}) = 0.5 \times 0.182 \times 0.364 = 0.033$ Prediction: Not Spam (0.033 > 0.015).

2.6 Implementation

python
Run Inline
# runnable
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.datasets import load_iris, fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Gaussian NB on Iris
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)
gnb = GaussianNB()
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
print(f"Gaussian NB accuracy: {accuracy_score(y_test, y_pred):.3f}")
# Multinomial NB for text (using 20 newsgroups subset)
categories = ['alt.atheism', 'soc.religion.christian']
newsgroups = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)
# Convert text to bag-of-words
vectorizer = CountVectorizer(stop_words='english', max_features=1000)
X_bow = vectorizer.fit_transform(newsgroups.data)
y = newsgroups.target
X_train, X_test, y_train, y_test = train_test_split(X_bow, y, test_size=0.3, random_state=42)
mnb = MultinomialNB(alpha=1.0)
mnb.fit(X_train, y_train)
y_pred = mnb.predict(X_test)
print(f"Multinomial NB accuracy: {accuracy_score(y_test, y_pred):.3f}")

📝 Practice Questions

Q1: Why is the "naive" assumption called naive?

Because it assumes conditional independence of features given the class — that all features are independent when we know the class. In reality, features are almost always correlated (e.g., "Houston" and "Texans" in spam emails). It's "naive" because it makes this strong assumption despite knowing it's usually false. Q2: What is Laplace smoothing and why is it needed?

Laplace smoothing adds a small constant (usually 1) to all count estimates to avoid zero probabilities. Without it, if a word never appeared in training for class "Spam," $P(word|Spam) = 0$, and the entire product becomes 0 regardless of other words. Smoothing prevents this. Q3: Why does Naive Bayes work well for text classification despite the independence assumption?

1. For argmax (which class to choose), only the ranking of probabilities matters, not the exact values
2. Even if feature dependencies exist, the relative ordering often remains correct
3. The independence assumption reduces variance (fewer parameters to estimate) — important with limited text data
4. For most text tasks, presence/absence of words is strongly informative despite correlations Q4: Compare Gaussian NB and Logistic Regression.