Orchestration: Kubernetes & Kubeflow — Scaling ML Pipelines

🎯 Learning Objectives

• Understand Kubernetes concepts for ML workload orchestration
• Deploy and scale ML models on Kubernetes
• Build Kubeflow pipelines for end-to-end ML workflows
• Manage GPU resources with Kubernetes
• Implement auto-scaling for model serving

📋 Prerequisites

• Docker/Containers (Week 5): Container images
• Basic Linux: Command line, SSH
• MLOps Lifecycle (Week 1): Understanding of ML pipeline stages

1. 📖 Core Content

1.1 Intuition: Why Orchestration?

• Different amounts of CPU/RAM/GPU
• To scale up during peak hours (10× traffic)
• Automatic recovery if a server crashes
• Rolling updates without downtime Managing servers manually for this is impossible. Kubernetes (K8s) automates it. Kubernetes = "Helmsman" (Greek) — it steers your containerized applications.

1.2 Kubernetes Core Concepts

1.3 Deploying an ML Model on Kubernetes

1.3.1 Deployment Manifest

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-detection
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fraud-detection
  template:
    metadata:
      labels:
        app: fraud-detection
    spec:
      containers:
      - name: model
        image: myrepo/fraud-model:v1.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
        env:
        - name: MODEL_PATH
          value: "/models/fraud_model.pkl"

1.3.2 Service Manifest (Exposing the Model)

yaml
apiVersion: v1
kind: Service
metadata:
  name: fraud-detection-service
spec:
  selector:
    app: fraud-detection
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

1.4 GPU Management in Kubernetes

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-model
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: model
        image: myrepo/gpu-model:v1
        resources:
          limits:
            nvidia.com/gpu: 1  # Request 1 GPU
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-tesla-t4  # Specific GPU type

1.5 Auto-Scaling ML Models

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: fraud-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: fraud-detection
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

1.6 Kubeflow: ML Platform on Kubernetes

1.6.1 Kubeflow Pipeline Example

python
Run Inline
from kfp import dsl
import kfp.components as comp
@dsl.component
def load_data(url: str) -> str:
    import pandas as pd
    df = pd.read_csv(url)
    output_path = '/data/dataset.csv'
    df.to_csv(output_path)
    return output_path
@dsl.component
def train_model(data_path: str) -> str:
    import pandas as pd
    from sklearn.ensemble import RandomForestClassifier
    import joblib
    df = pd.read_csv(data_path)
    X, y = df.drop('target', axis=1), df['target']
    model = RandomForestClassifier(n_estimators=100)
    model.fit(X, y)
    model_path = '/models/model.pkl'
    joblib.dump(model, model_path)
    return model_path
@dsl.component
def evaluate_model(model_path: str, data_path: str) -> float:
    import pandas as pd
    from sklearn.metrics import accuracy_score
    import joblib
    df = pd.read_csv(data_path)
    model = joblib.load(model_path)
    X, y = df.drop('target', axis=1), df['target']
    accuracy = accuracy_score(y, model.predict(X))
    return accuracy
@dsl.pipeline(
    name='ML Pipeline',
    description='End-to-end ML pipeline'
)
def ml_pipeline(data_url: str = 'https://example.com/data.csv'):
    load_task = load_data(url=data_url)
    train_task = train_model(data_path=load_task.output)
    eval_task = evaluate_model(
        model_path=train_task.output,
        data_path=load_task.output
    )

1.7 Edge Cases & Gotchas

• Pod startup time: Large models can take 5+ minutes to load. Set initialDelaySeconds high enough for readiness probes. Use init containers for pre-loading.
• GPU fragmentation: Single GPU can't be shared across pods (by default). Use MIG (Multi-Instance GPU) for partitioning if supported.
• Node affinity for GPU: If your model requires specific hardware (GPU type, CPU architecture), set node selector or affinity rules.
• Model caching: Pre-load models on a PersistentVolume to avoid downloading weights on every pod start.
• Resource limits: ML models can have memory spikes during inference. Set appropriate RAM limits and test with peak load.

1.8 Why This Matters

• Uber: Michelangelo runs on Kubernetes
• Netflix: Container orchestration for ML
• Spotify: Kubeflow for ML pipelines
• Google: Vertex AI is built on Kubeflow If you plan to deploy ML models at scale, you will encounter Kubernetes. Understanding the basics of K8s orchestration is essential for MLOps.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Requesting Too Many Resources

Pitfall 2: Not Setting Readiness Probes

Pitfall 3: Using Default Namespace

4. 📝 Practice Questions

Q1: Your model serving pod takes 45 seconds to load the model. The readiness probe has initialDelaySeconds=10. Explain the problem and fix.

Problem: Traffic is sent to the pod after 10 seconds (when the first probe runs). If the probe passes before the model loads (it might check a general endpoint), traffic arrives before the model is ready → 502 errors.

If the probe checks /health which only returns 200 after model loading, the pod is marked ready 10 seconds after loading completes, meaning traffic arrives 55+ seconds after pod creation.

Fix:

• Set initialDelaySeconds: 60 — wait 60 seconds before first probe
• Or reduce model loading time (e.g., use --no-daemon for model download before starting uvicorn)
• Or use init containers to pre-load model before the serving container starts Q2: A Kubernetes cluster has 4 nodes, each with 4 vCPU and 16 GB RAM. Each model pod needs 0.5 vCPU and 2 GB RAM. If 8 model pods are already running, can you deploy 4 more?

Total capacity: 4 nodes × 4 vCPU = 16 vCPU, 4 × 16 GB = 64 GB RAM.

8 pods: 8 × 0.5 = 4 vCPU, 8 × 2 = 16 GB RAM. Available: 12 vCPU, 48 GB RAM.

4 more pods: 4 × 0.5 = 2 vCPU, 4 × 2 = 8 GB RAM. After: 6 vCPU used, 24 GB RAM used. Available: 10 vCPU, 40 GB RAM.

Yes, you can deploy 4 more. The cluster would be at 37.5% utilization, suggesting you should either:

1. Reduce the number of nodes (cost savings)
2. Increase deployment replicas to handle more traffic Q3: Compare serverless deployment (AWS Lambda) vs Kubernetes for ML model serving. When would you choose each?

Choose Kubernetes when: You need GPU, have large models (1+ GB), need persistent endpoints, or require complex routing.

Choose Serverless when: You have small models (< 250 MB), CPU-only inference, sporadic traffic patterns, or want minimal ops overhead.

Many production systems use both: Kubernetes for GPU-intensive models, serverless for light-weight preprocessing/fallback models.

5. 🔗 Cross-References

• Previous: Containers & Docker (Week 5) — Containerizing models for K8s
• Next: CI/CD for ML (Week 7) — Automated deployment to K8s
• Related: Model Serving (Week 8) — Serving patterns on K8s
• External: Kubernetes documentation (kubernetes.io)
• External: Kubeflow documentation (kubeflow.org) Join Discord PreviousContainers & DockerNextCI/CD for ML