Model Monitoring: Data Drift, Concept Drift, and Observability

🎯 Learning Objectives

• Detect data drift and concept drift in production models
• Set up monitoring dashboards and alerts
• Implement automated retraining pipelines
• Understand ML observability best practices

📋 Prerequisites

• Model deployment concepts
• Statistical hypothesis testing
• MLOps lifecycle understanding

---

1. 📖 Core Content

1.1 Why Monitor?

1.2 Types of Drift

1.3 Detection Methods

Population Stability Index (PSI)

$$PSI = \sum_{i=1}^n (P_i - Q_i) \cdot \ln(P_i / Q_i)$$

Statistical Tests

1.4 Monitoring Infrastructure

python
Run Inline
# Simplified monitoring framework
class ModelMonitor:
    def __init__(self, reference_data, model):
        self.reference_stats = self.compute_statistics(reference_data)
        self.model = model
        self.alerts = []
    def compute_statistics(self, data):
        return {
            'mean': data.mean(axis=0),
            'std': data.std(axis=0),
            'histograms': [np.histogram(data[:, i], bins=20) for i in range(data.shape[1])]
        }
    def check_drift(self, current_data):
        # PSI for each feature
        for i, (ref_hist, _) in enumerate(self.reference_stats['histograms']):
            cur_hist, _ = np.histogram(current_data[:, i], bins=20)
            # Normalize
            ref_pct = ref_hist / ref_hist.sum() + 1e-10
            cur_pct = cur_hist / cur_hist.sum() + 1e-10
            psi = np.sum((ref_pct - cur_pct) * np.log(ref_pct / cur_pct))
            if psi > 0.2:
                self.alerts.append(f"Feature {i}: PSI={psi:.3f} — significant drift!")
        return self.alerts

1.5 Automated Retraining

python
Run Inline
def automated_retrain_pipeline():
    """CRON job: daily retrain if drift detected"""
    # 1. Get production data (last 7 days)
    prod_data = get_production_data(days=7)
    # 2. Check for drift
    alerts = monitor.check_drift(prod_data)
    if len(alerts) > 0:
        # 3. Collect new labeled data
        new_data = collect_and_label(prod_data)
        # 4. Retrain
        model = train_model(old_model, new_data)
        # 5. Validate
        if validate_model(model, test_data) > threshold:
            # 6. Deploy
            deploy_model(model)
            trigger_canary_rollout()
        # 7. Notify
        send_alert("Model retrained and deployed")

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1

<strong>Q1</strong>: A fraud detection model trained in 2022 performs well in testing but poorly in 2024. What type of drift is this?

This is likely concept drift: the relationship between features and fraud label has changed. Fraud patterns evolve — criminals adapt to detection methods. What looked like fraud in 2022 may now be different.

Evidence for concept drift (vs data drift):

• Feature distributions may be similar (users behave similarly)
• But the model's predictions are less accurate
• Fraudulent patterns have evolved (new types of attacks)

Data drift would show changes in feature distributions (e.g., transaction amounts increased). Concept drift shows changes in the decision boundary (what constitutes fraud changed).

For fraud detection, concept drift is expected and models typically need retraining every 1-6 months. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: PSI = 0.35 for a feature. Is this actionable? What threshold would you set?

PSI > 0.2 is typically considered "significant" drift. PSI = 0.35 is clearly actionable — the feature distribution has changed substantially.

Interpretation guidelines:

• PSI < 0.1: No significant change
• 0.1 ≤ PSI < 0.2: Moderate change, monitor
• PSI ≥ 0.2: Significant change, investigate
• PSI ≥ 0.3: Severe change, take action

For PSI = 0.35: trigger investigation. Which feature is it? Why did it change? Does it correlate with performance degradation? Should we retrain?

Action thresholds depend on your risk tolerance. High-stakes applications (fraud, healthcare, finance) should use lower thresholds (0.15 or 0.2). Low-stakes applications might tolerate PSI up to 0.4 before retraining. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3</strong>

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: In an ML system, you can't access ground truth labels in real-time. How do you monitor model performance?

Without real-time labels, you can't directly measure accuracy. Use proxy metrics:

1. Prediction drift: Monitor model output distribution. If the model suddenly predicts 80% "fraud" (previously 2%), something changed.
2. Feature drift: Monitor input distributions. Drift in features correlated with your target.
3. Heuristic fallback: Compare predictions against simple heuristics or rules.
4. Delayed labeling: Use day-lag analysis. Compare today's predictions against labels that arrive tomorrow/week.
5. Synthetic data: Create test cases with known outcomes; run continuously.
6. Human-in-the-loop: Randomly sample predictions for human review (especially important for high-stakes applications).
7. Statistical checks: Calibration curves, prediction confidence distribution, feature importance stability.

Best practice: combine multiple signals. Feature drift + prediction drift + periodic sampling gives high confidence even without real-time labels. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4</strong>

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: A model is retrained weekly. After each retrain, accuracy is 95%. By day 6, it drops to 88%. What does this suggest?

Sudden concept drift: Performance degrades rapidly within each week, suggesting a sudden change in the target concept rather than gradual drift.

Possible causes:

1. Weekly pattern: User behavior changes on weekends vs weekdays
2. Data freshness: The training batch is one week old and doesn't reflect current patterns
3. Competitor response: Fraudsters/criminals adapt within days of deployment

Solutions:

1. Daily retraining: More frequent updates
2. Exponential weighting: Weight recent data more heavily in training
3. Ensemble approach: Keep ensemble of models from different training windows
4. Online learning: Incrementally update model with each new batch of data
5. Day-of-week features: Include day-of-week or time-based features so model can capture periodic patterns

The specific pattern (weekly degradation) strongly suggests modeling time as a feature would help.

</details> * * * ## 🔗 Cross-References - **Next**: [A/B Testing](/notes/04-degree-electives-bsda5014-mlops-week10-10-ab-testing) - **Previous**: [Model Serving](/notes/04-degree-electives-bsda5014-mlops-week08-08-model-serving) - **Video**: BSDA5014 Week 9 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Model Serving**](/notes/04-degree-electives-bsda5014-mlops-week08-08-model-serving)[Next**A/B Testing & Shadow Deployment**](/notes/04-degree-electives-bsda5014-mlops-week10-10-ab-testing)