A/B Testing & Shadow Deployment: Canary Releases and Rollout Strategies

🎯 Learning Objectives

• Design A/B tests to compare ML model performance
• Implement shadow deployment patterns (mirroring traffic)
• Execute canary releases with progressive rollout
• Apply statistical significance testing for model comparison
• Understand rollback strategies for failed deployments

📋 Prerequisites

• MLOps Lifecycle (Week 1): Deployment and monitoring stages
• Basic Statistics: Hypothesis testing, p-values, confidence intervals
• Model Serving (Week 8): Serving infrastructure

1. 📖 Core Content

1.1 Intuition: Why A/B Test ML Models?

• Statistical noise: The test set is small, and the difference isn't significant
• Distribution shift: The production data distribution differs from the test set
• Business metric mismatch: AUC improved, but revenue didn't
• Latency cost: The new model is 2× slower, causing user experience degradation A/B testing in production is the only way to know if a model actually improves business metrics. It compares the new model (treatment) against the current model (control) with real traffic.

1.2 A/B Testing Framework for ML

1.2.1 Key Steps

1. Choose primary metric: Business-relevant and measurable (revenue per user, CTR, retention)
2. Form hypothesis: "New model increases click-through rate by at least 1%"
3. Calculate sample size: Use power analysis to determine minimum sample size
4. Random assignment: Split users into control (old model) and treatment (new model)
5. Run experiment: Collect data for predetermined duration (avoid peeking)
6. Analyze: Compute p-value, confidence intervals, practical significance
7. Decision: Rollout if statistically significant and practically meaningful

1.3 Shadow Deployment

• Control model (production): Serves predictions → users
• Shadow model (candidate): Receives same inputs → logs predictions → NO real user impact (Diagram) Benefits:

1. Zero risk — users never see shadow model's predictions
2. Can compare metrics on identical traffic
3. No statistical sample size limitations
4. Test latency, memory, and throughput under production load Limitations:

• Requires double the compute resources (run both models)
• Can't measure actual business impact (shadow model doesn't affect user experience)
• Need to ensure shadow model doesn't cause side effects (database writes, user notifications)

1.4 Canary Release

1.4.1 Canary Implementation with Kubernetes

yaml
# Canary deployment (5% traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-v2-canary
spec:
  replicas: 1  # 1 pod vs 19 pods for v1 = 5% traffic
  template:
    spec:
      containers:
      - name: model
        image: myrepo/model:v2
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: model-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"  # 5% traffic
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - backend:
          service:
            name: model-v2-service
          port: 80

Worked Example 1: Canary Rollout Decision

• Error rate: 0.05% (v1: 0.04%, within threshold)
• p99 latency: 95ms (v1: 90ms, within threshold)
• Conversion rate: 3.2% (v1: 3.1%, +3.2% improvement) Step 2: Advance to 25% traffic. Observe for 48 hours. Observed metrics:
• Error rate: 0.06% (acceptable)
• Conversion rate: 3.25% (v1: 3.1%, +4.8% improvement)
• p95 confidence interval: [3.1%, 3.4%] — improvement is significant Step 3: Advance to 50%, then 100%. Decision: Full rollout approved. Model v2 is better.

1.5 Statistical Significance for ML A/B Tests

1.5.1 Key Metrics

1.5.2 Sample Size Calculation

python
Run Inline
# runnable
def min_sample_size(effect_size, alpha=0.05, power=0.8):
    """Calculate minimum sample size for A/B test."""
    from scipy import stats
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)
    # For proportion-based metric (conversion)
    p = 0.1  # baseline conversion rate
    p1 = p
    p2 = p * (1 + effect_size)
    n = ((z_alpha * (2 * p * (1 - p)) ** 0.5 +
          z_beta * (p1 * (1 - p1) + p2 * (1 - p2)) ** 0.5) ** 2) / (p2 - p1) ** 2
    return int(n)
# To detect 5% lift with baseline 10% conversion
print(min_sample_size(0.05))  # ~50,000 per group

1.6 Rollback Strategies

1.7 Edge Cases & Gotchas

• Peeking problem: Don't check results daily and stop early. Pre-register the experiment duration.
• Network effects: Users interact with each other (social network). Simple A/B assignment fails — use cluster-randomized design.
• Carryover effects: Showing a user a bad recommendation today affects their behavior tomorrow. Use washout periods.
• Novelty effect: Users might click more on any change. Run experiments long enough (2+ weeks).
• Interaction effects: Multiple A/B tests running simultaneously can interact. Use overlapping experiment frameworks.

1.8 Why This Matters

• Teams deploy models based on offline metrics that may not translate
• Bad models degrade user experience and business metrics
• Good models may be incorrectly rejected due to statistical noise
• There's no objective way to compare competing approaches Top tech companies run thousands of A/B tests annually. Understanding how to design and analyze them is essential for production ML.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Peeking at Results and Stopping Early

Pitfall 2: Multiple Metrics Without Correction

• Choose one primary metric before the experiment
• Use Bonferroni correction for secondary metrics: α' = α / k
• Use the False Discovery Rate (FDR) for many metrics

Pitfall 3: Confusing Statistical and Practical Significance

4. 📝 Practice Questions

Q1: You run an A/B test comparing two recommendation models. Control (100K users) has 5.2% CTR. Treatment (100K users) has 5.5% CTR. Is this lift statistically significant? (α = 0.05)

Compute the z-statistic:

$\hat{p}_1 = 0.052$, $\hat{p}_2 = 0.055$, $n_1 = n_2 = 100000$

$\hat{p} = \frac{0.052 + 0.055}{2} = 0.0535$

$SE = \sqrt{0.0535 \times 0.9465 \times (1/100000 + 1/100000)} = \sqrt{0.000001013} = 0.001006$

$z = \frac{0.055 - 0.052}{0.001006} = 2.98$

Critical value for α = 0.05: z = 1.96.

Since 2.98 > 1.96, the lift is statistically significant (p < 0.01).

The 95% CI for the lift: $(0.003) \pm 1.96 \times 0.001006 = [0.001, 0.005]$

The lift is between 0.1% and 0.5% absolute (1.9% to 9.6% relative). It's statistically significant, but you should check if this lift is practically meaningful for your business. Q2: Your shadow deployment shows the new model has 2ms higher latency (p99: 52ms vs 50ms). Is this acceptable? How would you decide?

1. Check SLA: If your SLA is p99 < 100ms, 52ms is well within it. The 2ms difference is acceptable.
2. Business impact: Does added latency reduce user engagement? Amazon found every 100ms of latency costs 1% in revenue. At 2ms, the impact is ~0.02% — negligible.
3. Tradeoff: If the new model provides even a tiny accuracy improvement (say 0.5% better CTR), it easily outweighs the 2ms latency cost.
4. Monitor: If latency continues to increase under full load (the shadow test was at partial traffic), the actual latency might be higher.

Decision: Acceptable. Proceed to canary if accuracy/CTR improvement is meaningful. Q3: Design an A/B test for a fraud detection model. The new model catches 5% more fraud but costs 10% more per transaction (compute cost). What metrics would you use for the decision?

Primary metric: Net savings = Fraud savings - Compute cost

Secondary metrics:

1. False positive rate: New model might flag more legitimate transactions (user friction)
2. Fraud capture rate: 5% improvement confirmed?
3. Cost per transaction: 10% increase confirmed?
4. User complaints: More legitimate transactions blocked?

Decision framework:

• If net savings > 0: Rollout (the fraud savings outweigh compute costs)
• If net savings ≤ 0: Don't rollout (unless false positive rate also improved)

Example calculation:

• Baseline: $100K fraud/month,$10K compute costs = $90K net savings
• New model: $105K fraud caught (5$11K compute (10% more) = $94K net savings
• Improvement: $4K/month. Worth deploying.

But also check: if false positive rate increases from 1% to 3%, the user friction cost might outweigh the $4K savings.

5. 🔗 Cross-References

• Previous: Model Monitoring (Week 9) — Metrics feeding into A/B decisions
• Related: Model Serving (Week 8) — Serving infrastructure for canary
• Related: CI/CD for ML (Week 7) — Automated deployment pipeline
• External: Kohavi et al., "Trustworthy Online Controlled Experiments" — A/B testing at scale Join Discord PreviousModel MonitoringNextBSDA5014 — Machine Learning Operations (MLOps)