Evaluation Metrics: Accuracy, Precision, Recall, F1, ROC-AUC, RMSE, MAE
383 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Evaluation Metrics: Accuracy, Precision, Recall, F1, ROC-AUC, RMSE, MAE ## 🎯 Learning Objectives - Choose the right metric for classification and regression problems - Interpret confusion matrices and ROC curves - Handle class imbalance in metric selection - Understand the limitations of each metric ## 📖 Core Co...
Evaluation Metrics: Accuracy, Precision, Recall, F1, ROC-AUC, RMSE, MAE
🎯 Learning Objectives
- Choose the right metric for classification and regression problems
- Interpret confusion matrices and ROC curves
- Handle class imbalance in metric selection
- Understand the limitations of each metric
📖 Core Content
1.1 Classification Metrics
| Metric | Formula | Best For | Pitfall |
|---|---|---|---|
| Accuracy | (TP+TN)/(TP+TN+FP+FN) | Balanced classes | Misleading for imbalanced |
| Precision | TP/(TP+FP) | Minimize false positives | Spam detection |
| Recall | TP/(TP+FN) | Minimize false negatives | Medical screening |
| F1 | 2·P·R/(P+R) | Imbalanced classes | Treats P and R equally |
| ROC-AUC | Area under TPR vs FPR curve | Ranking quality | Insensitive to class probability |
python# runnable from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, confusion_matrix) import numpy as np # Example: fraud detection (1000 samples, 5% fraud) y_true = np.zeros(1000) y_true[:50] = 1 # 50 fraud cases y_pred = np.zeros(1000) y_pred[:30] = 1 # 30 predicted fraud (20 TP, 10 FP) print(f"Accuracy: {accuracy_score(y_true, y_pred):.4f}") # (20+940)/1000 = 0.960 print(f"Precision: {precision_score(y_true, y_pred):.4f}") # 20/(20+10) = 0.667 print(f"Recall: {recall_score(y_true, y_pred):.4f}") # 20/50 = 0.400 print(f"F1: {f1_score(y_true, y_pred):.4f}") # 2*0.667*0.4/(1.067) = 0.500
1.2 Regression Metrics
| Metric | Formula | Unit | Best For |
|---|---|---|---|
| MAE | $\frac{1}{n}\sum\ | y_i - \hat{y}_i\ | $ |
| MSE | Squared of target | Penalizes large errors | |
| RMSE | Same as target | Interpretation of MSE | |
| R² | Unitless (0-1) | Relative improvement |
1.3 Why This Matters
The right metric aligns the ML objective with the business goal. A fraud model optimized for accuracy (which would predict "not fraud" for everything and get 95% accuracy) misses all fraud cases. Choosing Recall or F1 matches the business need.
2. 📝 Practice Questions
Q1: A medical test detects a rare disease (0.1% prevalence). The model has 99% accuracy. Should you deploy it?No! 99% accuracy is misleading. With 0.1% prevalence:
- In 100K people: 100 have the disease, 99,900 don't
- 99% accuracy: 99 correctly identified diseased + 98,901 correctly identified healthy = 99,000 correct
- 1% error: 999 false positives, 1 false negative
The model will flag 998 healthy people as diseased (999 FP - 1 FN math correction): Actually with 99% accuracy, the model correctly identifies 99 diseased people but misses 1, and correctly identifies 98,901 healthy but flags 999 as diseased. Your precision is 99/(99+999) = 9% — only 9% of positive predictions are correct.Use precision-recall curve and choose thresholds based on cost of false positives vs false negatives. Join Discord PreviousFeature EngineeringNextCross-Validation