Model Serving: APIs, Containers, and Inference Optimization

🎯 Learning Objectives

• Deploy ML models as REST/gRPC APIs
• Use model serving frameworks (TorchServe, Triton)
• Optimize inference performance
• Implement A/B testing and shadow deployment

📋 Prerequisites

• Docker basics
• REST API concepts
• Model file formats (.pt, .onnx)

---

1. 📖 Core Content

1.1 Serving Architecture

1.2 Model Serving Options

1.3 Deployment Patterns

python
Run Inline
# FastAPI serving example
from fastapi import FastAPI
from pydantic import BaseModel
import torch
app = FastAPI()
model = torch.jit.load("model.pt").cuda()
model.eval()
class PredictionRequest(BaseModel):
    features: list[float]
class PredictionResponse(BaseModel):
    prediction: float
    probability: float
@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    with torch.no_grad():
        tensor = torch.tensor(request.features).unsqueeze(0).cuda()
        output = model(tensor)
        prob = torch.sigmoid(output).item()
        pred = 1 if prob > 0.5 else 0
    return PredictionResponse(prediction=pred, probability=prob)

1.4 Inference Optimization

---

📝 Practice Questions

Q1

<strong>Q1</strong>: A model takes 200ms per prediction on CPU. With batch_size=8, throughput increases to 500ms for 8 predictions. Compute throughput in predictions/second for both cases.

Single prediction: 1 / 0.2 = 5 predictions/sec Batched (8): 8 / 0.5 = 16 predictions/sec

Batching improves throughput by 3.2× even though latency per batch increases. The trade-off: single-request latency goes from 200ms to 500ms, but total system throughput improves.

This is why production systems batch requests and why latency-critical apps may prefer smaller batches. Q2

<strong>Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: What is cold start in serverless ML deployment, and how can it be mitigated?

Cold start: When a serverless function (AWS Lambda, GCP Cloud Run) hasn't been used recently, it must load the model from disk into memory before serving the first request. This can take 10-30 seconds.

Mitigations:

1. Keep warm: Periodic "ping" requests to keep the function loaded
2. Provisioned concurrency: Pre-allocate instances
3. Model optimization: Use smaller models, quantization (faster load)
4. Model caching: Store model in memory-mapped files or shared storage
5. Warm-up requests: Send a dummy request during deployment

For latency-critical applications, dedicated GPU instances (not serverless) are preferred. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: In shadow deployment, traffic is sent to both old and new models, but only the old model's response is returned to users. Why?

Shadow deployment:

1. New model processes requests but responses aren't shown to users
2. Results are logged and compared with the old model
3. If the new model performs well (accuracy, latency, error rates), it can be promoted

Purpose:

• Safe testing: No user-facing impact if new model fails
• Real-world validation: Test on actual traffic, not synthetic data
• Gradual confidence: Collect statistical evidence of improvement
• Rollback safety: Old model is always ready

Shadow deployment is the safest deployment strategy, followed by canary deployment and then full rollout. Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4

<strong>Q4</strong>: Compare A/B testing with canary deployment for model rollout.

Typical flow: A/B test to determine which model is better → Canary deployment of better model → Monitor → Full rollout.

A/B testing answers "which model should we use?" Canary deployment answers "how do we safely roll it out?"

</details> * * * ## 🔗 Cross-References - **Next**: [Monitoring](/courses/bsda5014/notes/.%2Fweek09%2F09-monitoring) - **Previous**: [CI/CD for ML](/notes/04-degree-electives-bsda5014-mlops-week07-07-cicd-ml) - **Video**: BSDA5014 Week 8 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**CI/CD for ML**](/notes/04-degree-electives-bsda5014-mlops-week07-07-cicd-ml)[Next**Model Monitoring**](/notes/04-degree-electives-bsda5014-mlops-week09-09-model-monitoring)