ML System Design
377 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# ML System Design ## 🎯 Learning Objectives - Design end-to-end ML systems for production - Choose between batch and real-time serving - Address data distribution shifts (monitoring, retraining) - Handle large-scale ML infrastructure decisions ## 📖 Core Content ### 9.1 ML System Components *(Diagram)* ### 9.2 Key...
ML System Design
🎯 Learning Objectives
- Design end-to-end ML systems for production
- Choose between batch and real-time serving
- Address data distribution shifts (monitoring, retraining)
- Handle large-scale ML infrastructure decisions
📖 Core Content
9.1 ML System Components
(Diagram)
9.2 Key Design Decisions
| Decision | Option 1 | Option 2 | Tradeoff |
|---|---|---|---|
| Serving | Batch (offline) | Real-time (online) | Latency vs recency |
| Training | Periodic retrain | Online learning | Freshness vs stability |
| Features | Pre-computed | On-the-fly | Storage vs compute |
| Model | Single model | Ensemble | Simplicity vs accuracy |
| Infrastructure | Managed (SageMaker) | Custom (Kubernetes) | Cost vs flexibility |
9.3 Batch vs Real-time Serving
| Aspect | Batch | Real-time |
|---|---|---|
| Latency | Hours to days | Milliseconds |
| Use Case | Recommendations (nightly) | Fraud detection (instant) |
| Compute | Predict all at once | Predict on-demand |
| Cost | Lower (can schedule) | Higher (always ON) |
| Freshness | Stale (last batch) | Current |
| Complexity | Simpler | Complex (scaling, latency) |
9.4 Model Monitoring
Drift types to monitor:
- Data drift: Input distribution changes (customers get older)
- Concept drift: Relationship between X and y changes (fashion trends)
- Prediction drift: Output distribution changes
- Target drift: Ground truth distribution changes Monitoring metrics:
- Data distribution statistics (mean, std per feature)
- Model performance (when labels available)
- Prediction volume and latency
- Feature importance over time
📝 Practice Questions
Q1: How would you design a real-time fraud detection system?
- Data pipeline: Kafka for transaction stream → feature computation in Flink
- Feature store: Redis for real-time features (user history, device fingerprints)
- Model: Lightweight ensemble (XGBoost + Neural Network) served via Docker on Kubernetes
- Inference: REST API with autoscaling (handle 10k QPS)
- Monitoring: Track prediction distribution, alert on drift, holdout for label feedback
- Retraining: Hourly retraining with latest confirmed fraud cases Q2: When would you choose batch serving over real-time?
Batch serving is better when: (1) predictions don't need to be instant (daily recommendations), (2) you need to process millions of predictions efficiently, (3) complex features require heavy computation, (4) cost is a primary concern. Example: Netflix generates recommendations nightly (batch) rather than recomputing every page load. Q3: What is the difference between data drift and concept drift?Data drift: P(X) changes. The input distribution shifts but the relationship P(y|X) remains the same. Example: users become older (age distribution shifts) but age→purchase relationship unchanged. Concept drift: P(y|X) changes. The mapping from input to output changes. Example: what was fashionable last year (input) now predicts "unpopular" (output). Join Discord PreviousSemi-supervised LearningNextDeep Learning Basics