Anomaly Detection

🎯 Learning Objectives

• Identify use cases for anomaly detection (fraud, monitoring, quality control)
• Implement statistical outlier detection (Z-score, IQR)
• Use Isolation Forest and LOF for multi-dimensional anomaly detection
• Evaluate anomaly detection performance

📖 Core Content

6.1 Intuition: Finding the Needle in the Haystack

6.2 Methods

• Z-score: $z = \frac{x - \mu}{\sigma}$, flag if $|z| > 3$
• IQR: Flag if $x < Q1 - 1.5 \times IQR$ or $x > Q3 + 1.5 \times IQR$
• Grubbs' test: Statistical test for a single outlier Density-Based Methods:
• LOF (Local Outlier Factor): Compares density around a point to density around its neighbors
• DBSCAN: Points labeled as -1 (noise) are anomalies Model-Based Methods:
• Isolation Forest: Randomly split features — anomalies are isolated in fewer splits
• One-Class SVM: Finds a boundary around normal data
• Autoencoder: High reconstruction error = anomaly

python
Run Inline
# runnable
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
import numpy as np
# Generate normal data with some outliers
np.random.seed(42)
X_normal = np.random.randn(200, 2) * 0.5
X_outliers = np.random.uniform(low=-3, high=3, size=(10, 2))
X = np.vstack([X_normal, X_outliers])
# Isolation Forest
iso_forest = IsolationForest(contamination=0.05, random_state=42)
y_pred_iso = iso_forest.fit_predict(X)
print(f"Isolation Forest anomalies: {np.sum(y_pred_iso == -1)}")
# LOF
lof = LocalOutlierFactor(contamination=0.05)
y_pred_lof = lof.fit_predict(X)
print(f"LOF anomalies: {np.sum(y_pred_lof == -1)}")

📝 Practice Questions

Q1: Why is anomaly detection harder than standard classification?

1. Rare events: Anomalies are < 1% of data — classifiers predict majority class and get 99% accuracy
2. No labels: Most anomaly detection is unsupervised (we don't know which points are anomalies)
3. Evolving anomalies: Fraud patterns change constantly
4. High cost of false positives: Too many alerts → alert fatigue Q2: How does Isolation Forest detect anomalies?

Isolation Forest randomly selects a feature and split value, creating binary partitions. Anomalies are few and different — they get isolated in fewer splits (shorter path length in the tree). Normal points require many splits to separate. The anomaly score is based on average path length across the forest. Q3: What contamination parameter means and how to set it?

Contamination is the expected proportion of anomalies in the data. It sets the threshold for flagging points. If you know fraud is 2% of transactions, set contamination=0.02. If unknown, estimate from domain knowledge or use auto-tuning methods. Join Discord PreviousTime SeriesNextSemi-supervised Learning