Experiment Tracking: MLflow, W&B, Metrics Logging, Hyperparameter Sweeps

🎯 Learning Objectives

• Set up MLflow for experiment logging and tracking
• Log metrics, parameters, and artifacts during training
• Visualize and compare runs in the MLflow UI
• Run hyperparameter sweeps with optimal parameter search
• Use Weights & Biases for cloud-based experiment tracking

📋 Prerequisites

• PyTorch training loops (Weeks 1-2)
• Basic Python

1. 📖 Core Content

1.1 Intuition: Why Track Experiments?

• "Was that run with LR=0.001 or 0.0001?" 🤔
• "Which model checkpoint had 92% accuracy?"
• "I can't reproduce last week's results" Experiment tracking solves this by logging:
• Parameters: Learning rate, batch size, architecture
• Metrics: Loss, accuracy, F1 per epoch
• Artifacts: Model weights, plots, confusion matrices
• Environment: Python version, GPU type, library versions

1.2 MLflow Setup

python
Run Inline
# runnable
import mlflow
import mlflow.pytorch
# Set tracking URI (local or remote)
mlflow.set_tracking_uri("file:./mlruns")  # Local
# mlflow.set_tracking_uri("http://localhost:5000")  # Server
# Create or set experiment
mlflow.set_experiment("dl-practice-experiments")
# Log a run
with mlflow.start_run(run_name="resnet50_lr0.001"):
    # Log parameters
    mlflow.log_param("model", "resnet50")
    mlflow.log_param("learning_rate", 0.001)
    mlflow.log_param("batch_size", 64)
    mlflow.log_param("optimizer", "adam")
    # Log metrics during training
    for epoch in range(5):
        train_loss = 2.0 * 0.5 ** epoch
        val_acc = 0.5 + 0.1 * epoch
        mlflow.log_metric("train_loss", train_loss, step=epoch)
        mlflow.log_metric("val_accuracy", val_acc, step=epoch)
    # Log model artifact
    mlflow.pytorch.log_model(model, "model")
    # Log additional artifacts
    mlflow.log_artifact("training_plot.png")
    mlflow.log_artifact("confusion_matrix.png")

1.3 Hyperparameter Sweeps

python
Run Inline
# runnable
import itertools
import mlflow
# Define search space
param_grid = {
    'learning_rate': [0.1, 0.01, 0.001],
    'batch_size': [32, 64],
    'dropout': [0.0, 0.3]
}
# Grid search
best_acc = 0
best_params = None
results = []
for lr, bs, dropout in itertools.product(*param_grid.values()):
    with mlflow.start_run():
        # Log params
        mlflow.log_param("lr", lr)
        mlflow.log_param("batch_size", bs)
        mlflow.log_param("dropout", dropout)
        # Simulate training
        val_acc = 0.8 - 0.5 * abs(lr - 0.01) - 0.1 * dropout + 0.02 * (bs == 64)
        # Log metric
        mlflow.log_metric("val_accuracy", val_acc)
        results.append({
            'lr': lr, 'batch_size': bs, 'dropout': dropout,
            'val_accuracy': val_acc
        })
        if val_acc > best_acc:
            best_acc = val_acc
            best_params = {'lr': lr, 'batch_size': bs, 'dropout': dropout}
print(f"Best accuracy: {best_acc:.4f}")
print(f"Best params: {best_params}")

1.4 Weights & Biases (W&B) Quick Start

python
Run Inline
# runnable
# import wandb
# Initialize W&B
# wandb.init(project="dl-practice", name="resnet-experiment")
# Log config
# config = wandb.config
# config.learning_rate = 0.001
# config.architecture = "ResNet-50"
# Log metrics in training loop
# for epoch in range(10):
#     wandb.log({
#         "epoch": epoch,
#         "train_loss": train_loss,
#         "val_accuracy": val_acc,
#         "learning_rate": current_lr
#     })
# # Log images and plots
# wandb.log({"predictions": [wandb.Image(img, caption=pred)]})
# wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(...)})
# # Finish
# wandb.finish()
print("W&B integration (uncomment to use)")

1.5 Why This Matters

• Compare hundreds of runs systematically
• Share results with collaborators
• Rollback to best-performing models
• Debug training issues by comparing runs
• Automate hyperparameter optimization

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Not Logging the Right Things

Pitfall 2: Forgetting to Set the Seed

4. 📝 Practice Questions

Q1: You run 50 experiments with different hyperparameters. The best run has validation accuracy 0.87. How do you know if this is significantly better than the second-best (0.865)?

1. Check the seed: If the seed wasn't fixed, the 0.005 difference could be noise. Re-run the top 2 configurations with 3-5 different seeds each.
2. Check training curves: Did the best run actually converge, or did it get lucky on the validation split? Plot training curves.
3. Statistical significance: Run each configuration 5 times. Report mean ± std. If 0.87±0.01 vs 0.865±0.01, the difference isn't significant.
4. Effect size: 0.5% improvement may not be practically meaningful. Consider whether the added complexity of the "best" parameters is worth the tiny gain.

5. 🔗 Cross-References

• Previous: Model Deployment (Week 8)
• Next: NLP Projects (Week 10)
• Related: MLOps Experiment Tracking
• External: MLflow Documentation (mlflow.org) Join Discord PreviousModel DeploymentNextNLP Projects