Logistic Regression & Classification Metrics

🎯 Learning Objectives

• Explain why linear regression fails for classification and how logistic regression fixes it
• Understand the sigmoid function and how it produces probabilities
• Derive the logistic regression cost function and gradient
• Evaluate classification models with accuracy, precision, recall, F1, and AUC-ROC
• Implement logistic regression using sklearn

📋 Prerequisites

• Linear Regression — logistic regression is structurally similar but for classification
• Probability basics — odds, log-odds, probability
• Gradient descent — same optimization algorithm

📖 Core Content

4.1 Intuition: Why Not Just Use Linear Regression for Classification?

1. Predictions outside [0,1]: For very few hours, the line might predict -0.2 (meaningless as a probability). For many hours, it might predict 1.3.
2. Sensitive to outliers: Adding more data points far to the right would tilt the regression line, changing all predictions. Logistic regression fixes this by wrapping the linear model in the sigmoid function, which squashes outputs to the range [0,1] — interpretable as probabilities. (Diagram)

4.2 The Sigmoid Function

• Range: (0, 1) — never exactly 0 or 1
• $\sigma(0) = 0.5$ — the decision threshold
• As $z \to +\infty$, $\sigma(z) \to 1$
• As $z \to -\infty$, $\sigma(z) \to 0$

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
z = np.linspace(-10, 10, 100)
sigmoid = 1 / (1 + np.exp(-z))
plt.plot(z, sigmoid, 'b-', linewidth=2)
plt.axhline(y=0.5, color='r', linestyle='--', label='Decision boundary (0.5)')
plt.axvline(x=0, color='gray', linestyle=':', alpha=0.5)
plt.xlabel('z = θᵀx')
plt.ylabel('σ(z) = P(y=1|x)')
plt.title('Sigmoid Function')
plt.grid(True, alpha=0.3)
plt.legend()
plt.show()

4.3 Formal Definition

4.4 The Cost Function

• If $y=1$: cost = $-\log(h_\theta(x))$. If $h_\theta(x) \approx 1$, cost ≈ 0. If $h_\theta(x) \approx 0$, cost → ∞.
• If $y=0$: cost = $-\log(1 - h_\theta(x))$. If $h_\theta(x) \approx 0$, cost ≈ 0. If $h_\theta(x) \approx 1$, cost → ∞. The gradient (same form as linear regression!):

4.5 Worked Example 1: Hand Calculation

4.6 Multiclass Classification: Softmax

python
Run Inline
# runnable
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
iris = load_iris()
X, y = iris.data, iris.target  # 3 classes of iris flowers
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# One-vs-Rest (default) or multinomial
model = LogisticRegression(multi_class='multinomial', max_iter=200)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
probabilities = model.predict_proba(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Sample probabilities: {probabilities[0]}")
print(f"Predicted class: {y_pred[0]}")

4.7 Classification Evaluation Metrics

python
Run Inline
# runnable
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve
import numpy as np
y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 0])
y_pred = np.array([0, 1, 1, 1, 0, 1, 0, 0, 1, 0])
print("Confusion Matrix:")
print(confusion_matrix(y_true, y_pred))
print("\nClassification Report:")
print(classification_report(y_true, y_pred))

4.8 Worked Example 2: Computing Metrics

• Accuracy = (80+90)/200 = 170/200 = 0.85
• Precision = 80/(80+10) = 80/90 = 0.889
• Recall = 80/(80+20) = 80/100 = 0.80
• F1 = 2(0.889×0.80)/(0.889+0.80) = 2(0.711)/1.689 = 0.842 Interpretation:
• "85% of all emails were correctly classified"
• "When the model says spam, it's 88.9% correct"
• "The model catches 80% of actual spam"
• "F1 of 0.842 balances precision and recall"

4.9 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Using Accuracy on Imbalanced Data

Pitfall 2: Not Setting the Decision Threshold

Pitfall 3: Multicollinearity in Logistic Regression

📝 Practice Questions

Q1: Calculate σ(2) manually.

$\sigma(2) = \frac{1}{1 + e^{-2}} = \frac{1}{1 + 0.135} = \frac{1}{1.135} = 0.881$

The probability of class 1 is 88.1% when z=2. Q2: With θ = [-1, 0.5], what's the decision boundary equation?

Answer: The decision boundary is where $\theta^T x = 0$: $-1 + 0.5x = 0 \implies x = 2$

So if the feature value > 2, predict class 1 (< 2, predict class 0). Q3: For a cancer test: TP=90, FP=10, FN=10, TN=890 out of 1000 patients. Compute accuracy, precision, recall, and F1.

• Accuracy = (90+890)/1000 = 0.98 (98%)
• Precision = 90/(90+10) = 0.90 (90%)
• Recall = 90/(90+10) = 0.90 (90%)
• F1 = 2(0.9×0.9)/(0.9+0.9) = 0.90 (90%)

Note: Even with 98% accuracy, 10 out of 100 cancer patients are missed (FN=10). Is that acceptable? Q4: Why can't we use MSE for logistic regression?

Answer: MSE creates a non-convex cost function for logistic regression with many local minima. Gradient descent might get stuck in a suboptimal local minimum. Log loss (binary cross-entropy) is convex — it has a single global minimum that gradient descent reliably finds. Q5: In a spam filter, which matters more: precision or recall?

Answer: Precision typically matters more. A false positive (marking legitimate email as spam) is worse than a false negative (letting spam through). The user will be more annoyed by missing important emails than by seeing occasional spam. But this depends on the use case — in an exam pass/fail model, both matter equally. Q6: Convert odds of 4:1 to probability and log-odds.

Odds = P(y=1) / P(y=0) = 4/1 = 4

Probability: P = odds/(1+odds) = 4/5 = 0.80

Log-odds = ln(4) = 1.386

In logistic regression, $\theta^T x$ gives the log-odds. So $\theta^T x = 1.386$ corresponds to 80% probability. Q7: AUC = 0.5 means what about the model?

Answer: AUC = 0.5 means the model is no better than random guessing. The ROC curve follows the diagonal line — for every threshold, the true positive rate equals the false positive rate. The model has zero discriminatory power. Q8: Implement logistic regression for 2D Iris data using sklearn

python
Run Inline
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

iris = load_iris()
# Use only first 2 features for binary classification (setosa vs others)
X = iris.data[:100, :2]  # Only setosa (0) and versicolor (1)
y = iris.target[:100]  # Binary: 0 or 1

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test):.3f}")
print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_}")

Q9: What is the difference between one-vs-rest and multinomial (softmax) for multiclass?

Answer:

• One-vs-Rest (OvR): Trains K separate binary classifiers (one per class vs. all others). Each gets its own set of parameters. Simple but probabilities may not sum to 1.
• Softmax/Multinomial: A single model with K sets of parameters. Produces a proper probability distribution (all classes sum to 1). Preferred when classes are mutually exclusive. Q10: Your model has high precision but low recall. What does this mean?

Answer:

• High precision: When the model predicts positive, it's usually right (few false positives).
• Low recall: The model misses many actual positives (many false negatives).

The model is very "cautious" — it only predicts positive when it's very confident, so it misses borderline cases. To improve recall, lower the decision threshold below 0.5. Q11: When would F1 be more appropriate than accuracy?

Answer: F1 is better when classes are imbalanced. Example: 95% legitimate, 5% fraudulent transactions. A model predicting "legitimate" always gets 95% accuracy. F1 captures the model's ability to detect the minority (fraud) class. Always use F1 (or precision/recall) when the positive class < 20% of the data. Q12: Explain the confusion matrix for a COVID test with 1% prevalence, test sensitivity 95%, specificity 99%. Compute PPV.

Out of 10,000 people: 100 sick (TP=95, FN=5), 9900 healthy (TN=9801, FP=99).

PPV (Precision): TP/(TP+FP) = 95/(95+99) = 95/194 = 0.49

Interpretation: Even with 95% sensitivity and 99% specificity, only 49% of positive tests are actually sick! This is because the disease is rare (1% prevalence). This counterintuitive result is the "base rate fallacy."

🔗 Cross-References

• Next Topic: k-Nearest Neighbors — instance-based learning
• Related: Classification Metrics — deeper dive on ROC, PR curves
• Related: Gradient Descent — optimization details
• External: IITM BSCS2004 Week 4, Hands-On ML Ch. 4, ISLR Ch. 4 Join Discord PreviousMultiple & Polynomial RegressionNextk-Nearest Neighbors