DVC: Data Version Control for ML Projects

🎯 Learning Objectives

• Version datasets and models with DVC alongside Git
• Create reproducible ML pipelines with DVC stages
• Share data via remote storage (S3, GCS, HDFS)
• Track experiments and compare results

📖 Core Content

1.1 Why DVC?

• Storing large files in external storage (S3, GCS)
• Keeping lightweight pointer files in Git
• Versioning data alongside code (same commit)
• Building reproducible ML pipelines

bash
# Initialize DVC
git init
dvc init
# Track a dataset
dvc add data/train.csv
git add data/train.csv.dvc .gitignore
git commit -m "Add training data"
# Add remote storage
dvc remote add -d storage s3://my-bucket/dvc-store
dvc push  # Upload data to remote

1.2 DVC Pipelines

python
Run Inline
# dvc.yaml
stages:
  clean:
    cmd: python src/clean.py data/raw data/clean
    deps:
      - data/raw
      - src/clean.py
    outs:
      - data/clean
  train:
    cmd: python src/train.py data/clean models/model.pkl
    deps:
      - data/clean
      - src/train.py
    outs:
      - models/model.pkl
    metrics:
      - metrics.json:
          cache: false

bash
dvc repro     # Reproduce pipeline
dvc metrics show  # Show metrics
dvc metrics diff  # Compare with previous version

1.3 Why This Matters

2. 📝 Practice Questions

Q1: A colleague says they can't reproduce your model because "the data changed." How does DVC prevent this?

DVC stores a hash (MD5) of every tracked file. The .dvc file committed to Git contains this hash. When you dvc checkout, DVC verifies that the local data matches the hash. If someone changed the data, dvc checkout fails with a checksum mismatch.

Additionally, dvc repro only re-runs pipeline stages whose dependencies changed. If the data hash is the same as when the model was built, DVC skips retraining because the cached output is still valid.

This guarantees: given the same Git commit + dvc checkout, you always get exactly the same data and model as the original run. Join Discord PreviousCLI ToolsNextPandas & NumPy