CI/CD for ML: GitHub Actions, Jenkins, and Continuous Training

🎯 Learning Objectives

• Design CI/CD pipelines for ML projects
• Implement automated model training and testing with GitHub Actions
• Set up Continuous Training (CT) pipelines
• Validate models before production deployment
• Understand the differences between software CI/CD and ML CI/CD

📋 Prerequisites

• Git/GitHub basics: Repos, branches, PRs
• ML Pipeline concepts: Training, evaluation, deployment stages
• Docker basics (Week 5): Container images

1. 📖 Core Content

1.1 Intuition: Why CI/CD for ML?

1. Data changes can trigger retraining (not just code changes)
2. Model validation is more complex (not just pass/fail)
3. Two artifacts to version (code + model weights)
4. Training is expensive (hours vs minutes for software tests)

1.2 ML CI/CD Pipeline Stages

1.3 GitHub Actions for ML

1.3.1 Basic Training Pipeline

yaml
name: ML Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install Dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest pandas scikit-learn
      - name: Run Tests
        run: pytest tests/ -v
      - name: Train Model
        run: python src/train.py
      - name: Evaluate Model
        run: python src/evaluate.py
      - name: Upload Model Artifact
        uses: actions/upload-artifact@v3
        with:
          name: model
          path: models/model.pkl

1.3.2 Continuous Training with Scheduled Trigger

yaml
name: Continuous Training
on:
  schedule:
    - cron: '0 6 * * 1'  # Every Monday at 6 AM
  workflow_dispatch:  # Manual trigger
jobs:
  retrain:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Fetch Latest Data
        run: python scripts/fetch_latest_data.py
      - name: Train Model
        run: python src/train.py --data latest
      - name: Evaluate vs Production
        run: python src/compare_models.py \
          --new models/new_model.pkl \
          --prod models/prod_model.pkl
      - name: Promote if Better
        run: |
          if python src/check_improvement.py; then
            echo "Model improved! Promoting to production."
            python src/promote_to_prod.py
          else
            echo "No significant improvement."
          fi

1.4 Model Validation Gates

Worked Example 1: Model Evaluation Gate

python
Run Inline
# runnable
def validate_model(new_model, baseline_model, test_data):
    """Validate new model against baseline."""
    results = {}
    # Gate 1: Test set performance
    new_score = evaluate(new_model, test_data)
    baseline_score = evaluate(baseline_model, test_data)
    results['accuracy_gain'] = new_score - baseline_score
    results['accuracy_gate'] = results['accuracy_gain'] >= 0.01  # 1% improvement
    # Gate 2: Check for performance degradation on subsets
    for subgroup in ['low_income', 'high_income', 'young', 'old']:
        sub_data = test_data.filter(pl.col('group') == subgroup)
        new_sub_score = evaluate(new_model, sub_data)
        baseline_sub_score = evaluate(baseline_model, sub_data)
        results[f'fairness_{subgroup}'] = (new_sub_score / baseline_sub_score) > 0.95
    # Gate 3: Robustness to missing values
    corrupted = test_data.with_columns(pl.col('feature_1').fill_null(0))
    robust_score = evaluate(new_model, corrupted)
    clean_score = evaluate(new_model, test_data)
    results['robustness'] = (robust_score / clean_score) > 0.9
    # Overall gate
    all_gates_passed = (
        results['accuracy_gate'] and
        all(results[f'fairness_{g}'] for g in ['low_income', 'high_income', 'young', 'old']) and
        results['robustness']
    )
    return all_gates_passed, results

1.5 Continuous Training (CT)

1.5.1 When to Retrain?

1.6 CI/CD Tools Comparison

1.7 Edge Cases & Gotchas

• Training timeouts: Long training runs may exceed CI limits (GitHub Actions: 6 hours). Use self-hosted runners or separate training cluster.
• Data access: CI/CD runners may not have access to production databases. Use sampled data for tests, full data for training.
• GPU availability: Standard CI runners don't have GPUs. Use cloud runners (GitHub Actions + AWS/GCP) or self-hosted GPU runners.
• Model artifact size: Large models (5+ GB) aren't suitable for artifact storage. Use cloud storage (S3, GCS) with registry pointers.
• Credential management: Store API keys and secrets in GitHub Secrets / Jenkins Credentials, not in code.

1.8 Why This Matters

• Team collaboration: Automated checks before merging code
• Reproducibility: Every training run is logged and versioned
• Deployment velocity: Push models to production with confidence
• Compliance: Audit trail of model changes for regulated industries Without ML CI/CD, teams face "works on my machine" problems, manual deployment errors, and untracked model changes.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Treating ML CI/CD Like Software CI/CD

Pitfall 2: Not Versioning Data Alongside Code and Model

Pitfall 3: Deploying Models Without Shadow Testing

4. 📝 Practice Questions

Q1: Design a GitHub Actions pipeline that: (a) runs unit tests on every PR, (b) trains a model on PR merge to main, (c) deploys to production only if model improves.

yaml
name: ML CI/CD
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  # Job 1: Fast checks on every PR
  test:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with: { python-version: '3.9' }
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --timeout=120
      - run: python scripts/validate_data.py

  # Job 2: Train on push to main
  train:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - run: pip install -r requirements.txt
      - run: python src/train.py --output models/candidate.pkl

      # Compare with production
      - run: python src/compare.py --candidate models/candidate.pkl --prod prod/model.pkl

      # Deploy if better
      - name: Deploy Candidate
        if: steps.compare.outputs.improved == 'true'
        run: python scripts/deploy.py models/candidate.pkl

Q2: Your scheduled retraining pipeline ran, but the new model's accuracy dropped 5% on the validation set. What should happen?

The pipeline should:

1. Log the failure: Record the new model's metrics in MLflow alongside the decreased accuracy
2. Alert the team: Send notification (Slack/email) with summary: "Model v2.1 accuracy dropped from 92% to 87%"
3. Preserve current production model: Do NOT deploy the worse model
4. Investigate root cause: Check data drift (feature distributions), concept drift (relationship shift), data quality issues (missing values, labeling errors)
5. Trigger data pipeline: If data quality issue is detected, trigger data validation and correction
6. Option to retrain with old data: If data is corrupted, retrain on the last good dataset version

The automated check if accuracy_drop > threshold: block_deployment = True prevents automatic degradation. Q3: Compare the CI/CD costs for a model that trains in 30 minutes (CPU) vs 6 hours (GPU). How would each fit into typical CI/CD budgets?

CPU model (30 min):

• GitHub Actions: 30 min × $0.008/min (standard) =$0.24 per run
• Daily retrain: ~$7.20/month
• Weekly retrain: ~$1.00/month

Fits comfortably in any CI/CD budget. Can run multiple times per day.

GPU model (6 hours):

• GitHub Actions: 360 min × $0.08/min (GPU runner) =$28.80 per run
• Daily retrain: ~$864/month
• Weekly retrain: ~$115/month

Too expensive for daily GitHub Actions GPU runs. Alternatives:

1. Self-hosted GPU runner: One-time cost, lower per-run cost
2. Separate training cluster (AWS Batch, GCP AI Platform): Pay per instance, more flexible
3. Reduce frequency: Weekly retrain instead of daily
4. Hybrid: Train on cheap preemptible/spot instances, only use GPU CI for validation

For GPU training-heavy pipelines, most teams use a dedicated compute cluster rather than CI-integrated runners.

5. 🔗 Cross-References

• Previous: Orchestration: K8s & Kubeflow (Week 6) — Pipeline orchestration
• Next: Model Serving (Week 8) — Deploying validated models
• Related: Experiment Tracking (Week 2) — Logging runs during CI
• External: GitHub Actions documentation — Practical CI/CD examples Join Discord PreviousOrchestration: K8s & KubeflowNextModel Serving