Experiment Tracking with MLflow and W&B

🎯 Learning Objectives

• Set up MLflow/W&B for experiment logging
• Log parameters, metrics, and artifacts
• Implement hyperparameter sweeps
• Compare experiments and reproduce results

📋 Prerequisites

• Python ML workflow
• Basic understanding of model training

---

1. 📖 Core Content

1.1 Why Experiment Tracking?

1. Reproducibility: Exact record of all experiment details
2. Comparison: Side-by-side metric comparison
3. Organization: Search/filter through hundreds of runs
4. Collaboration: Share results with team
5. Documentation: Automatic experiment history

1.2 MLflow Tracking

python
Run Inline
# runnable
# MLflow tracking example
import mlflow
def train_with_mlflow():
    mlflow.set_experiment("my-experiment")
    with mlflow.start_run():
        # Log parameters
        mlflow.log_param("learning_rate", 1e-3)
        mlflow.log_param("batch_size", 64)
        mlflow.log_param("optimizer", "Adam")
        mlflow.log_param("model_type", "ResNet-50")
        # Training loop
        for epoch in range(10):
            train_loss = train_epoch()
            # Log metrics (per epoch)
            mlflow.log_metric("train_loss", train_loss, step=epoch)
            if epoch % 5 == 0:
                val_accuracy = validate()
                mlflow.log_metric("val_accuracy", val_accuracy, step=epoch)
        # Log artifacts
        mlflow.log_artifact("model.pt")
        mlflow.log_artifact("confusion_matrix.png")
        # Log model
        mlflow.pytorch.log_model(model, "model")

1.3 Key Concepts

1.4 Hyperparameter Sweeps

python
Run Inline
# MLflow sweep configuration
sweep_config = {
    "method": "bayesian",  # or "grid", "random"
    "metric": {"name": "val_accuracy", "goal": "maximize"},
    "parameters": {
        "learning_rate": {"min": 1e-4, "max": 1e-2},
        "batch_size": {"values": [32, 64, 128]},
        "dropout": {"min": 0.1, "max": 0.5},
        "hidden_dims": {"values": [128, 256, 512]}
    }
}

1.5 Best Practices

1. Log everything: Every hyperparameter, metric, and version
2. Use consistent naming: Project → experiment → run hierarchy
3. Log environment: Python version, package versions, GPU info
4. Log the data: Data version, split sizes, preprocessing
5. Tag important runs: "production", "baseline", "ablation"

---

📝 Practice Questions

Q1

<strong>Q1</strong>: Why log the Git commit hash with each experiment?

The Git commit hash uniquely identifies the exact code version used for the experiment. This enables:

1. Reproducibility: Revert to the exact code version and rerun
2. Debugging: If a change causes performance drop, trace back to the commit
3. Collaboration: Team members know which code produced which results
4. Auditing: Track who changed what and when

Without it, even with all hyperparameters logged, different code versions may produce different results. Q2

<strong>Q2

<strong>Q2</strong>: You have 100 training runs and need to find the best configuration. How would MLflow/W&B help?

Using the tracking UI:

1. Filter: Filter by experiment, tags, or parameters
2. Sort: Sort by val_accuracy descending
3. Compare: Select top runs and view side-by-side parameters
4. Parallel coordinates: Visualize which parameter combinations work best
5. Download: Export results as CSV for further analysis

Without a tracking tool, this would require manually collecting results from log files. Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: A colleague can't reproduce your model's accuracy. What tracking information would help?

1. Exact data version: Which dataset version was used?
2. Code version: Git commit hash
3. Environment: Python packages + versions (requirements.txt)
4. Hyperparameters: All training parameters logged
5. Random seed: So random operations are deterministic
6. GPU: Model may behave differently on different GPU architectures
7. Preprocessing: Any normalization/augmentation applied

Without this information, reproducing ML results is often impossible — this is the reproducibility crisis in ML.

</details> * * * ## 🔗 Cross-References - **Next**: [Data Versioning](/notes/04-degree-electives-bsda5014-mlops-week03-03-data-versioning-dvc) - **Video**: BSDA5014 Week 2 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**MLOps Lifecycle**](/notes/04-degree-electives-bsda5014-mlops-week01-01-mlops-lifecycle)[Next**Data Versioning**](/notes/04-degree-electives-bsda5014-mlops-week03-03-data-versioning-dvc)