Data & Model Versioning with DVC and Git-LFS

🎯 Learning Objectives

• Version datasets and models alongside code using DVC
• Implement DVC pipelines for reproducible ML workflows
• Use Git-LFS for storing large binary files
• Track data lineage from source to model

📋 Prerequisites

• Git version control basics
• Understanding of ML pipelines

---

1. 📖 Core Content

1.1 The Versioning Problem

1.2 DVC (Data Version Control)

bash
# Initialize DVC in Git repo
git init && dvc init
# Add data to DVC tracking
dvc add data/train.csv
# Creates data/train.csv.dvc (pointer file) + adds data/train.csv to .gitignore
# Commit pointer files to Git
git add data/train.csv.dvc .gitignore
git commit -m "add training data"
# Push data to remote storage
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push
# Later, reproduce the exact state
git checkout <commit>
dvc checkout  # Restores data files

1.3 DVC Pipelines

yaml
# dvc.yaml
stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - data/raw
    params:
      - prepare.split_ratio
    outs:
      - data/prepared
  train:
    cmd: python src/train.py
    deps:
      - data/prepared
      - src/train.py
    params:
      - train.lr
      - train.epochs
    outs:
      - models/model.pt
  evaluate:
    cmd: python src/evaluate.py
    deps:
      - models/model.pt
      - data/prepared
    metrics:
      - metrics/accuracy.json

1.4 Git-LFS

python
Run Inline
# .gitattributes
*.csv filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text

1.5 DVC vs Git-LFS

---

📝 Practice Questions

Q1

<strong>Q1

<strong>Q1</strong>: A colleague can't reproduce your model's accuracy. They have the same Git commit but different results. What might be wrong?

Possible causes:

1. Data version mismatch: Same Git commit but different DVC data version
2. Different data source: Raw data was updated but DVC wasn't run
3. Dependencies not tracked: A dependency (e.g., a config file) changed without Git tracking
4. Random seed: Random seed not set for reproducibility
5. Environment: Different package versions

Fix: Run dvc status to check if data/pipeline stages match. The commit should include params.yaml (tracked by Git) and data pointers (tracked by DVC). Full reproducibility requires:

• git checkout <commit>
• dvc checkout (restores data)
• dvc repro (reruns pipeline if needed)
• pip install -r requirements.txt (same dependencies) Q2

<strong>Q2

<strong>Q2

<strong>Q2</strong>: Why does DVC create .dvc pointer files instead of storing data directly in Git?

1. Repository size: Data files (GBs) would make Git repos huge and slow to clone
2. Binary diffs: Git can't efficiently diff binary data files
3. Storage backend flexibility: DVC can use S3, GCS, HDFS, or SSH — independent of Git hosting
4. Large file support: Git struggles with files >50MB
5. Data access control: Separate permissions for code and data

The .dvc file contains only the file hash (a 64-character string), which Git can handle efficiently. The actual data lives in DVC's cache. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: Describe a workflow for versioning an ML dataset that gets monthly updates.

sql
Month 1: data_v1.csv → dvc add → commit .dvc file
Month 2: data_v2.csv → dvc add → commit new .dvc file
Month 2 (fix): data_v2.1 → dvc add → commit .dvc file

Each version is independently accessible:
git checkout v1 → dvc checkout → get data_v1
git checkout v2 → dvc checkout → get data_v2

Or use branches:
git checkout -b data-v2
dvc add data.csv  # Updates pointer file
git commit -m "update data to v2"

This allows comparing model performance across data versions by checking out different commits and running dvc checkout to get the corresponding data.

</details> * * * ## 🔗 Cross-References - **Next**: [Feature Stores](/notes/04-degree-electives-bsda5014-mlops-week04-04-feature-stores) - **Previous**: [Experiment Tracking](/notes/04-degree-electives-bsda5014-mlops-week02-02-experiment-tracking) - **Video**: BSDA5014 Week 3 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Experiment Tracking**](/notes/04-degree-electives-bsda5014-mlops-week02-02-experiment-tracking)[Next**Feature Stores**](/notes/04-degree-electives-bsda5014-mlops-week04-04-feature-stores)