What does DVC store in Git?

DVC stores small metadata files, pipeline definitions and lock information in Git. The large files themselves, such as raw datasets, cleaned datasets, model outputs and intermediate artifacts, are kept in a DVC remote. A Git commit can then point to both the code and the exact data versions used with it.

How does DVC make preprocessing reproducible?

DVC records the inputs, commands, parameters and outputs for each pipeline stage. When reproduction runs, it compares the recorded state with the current project state. Unchanged stages can reuse existing artifacts, while changed dependencies or parameters cause only the affected work to run again.

Why not just commit processed datasets to Git?

Git is poor at handling large changing data files, and committing generated datasets directly makes repositories heavy and awkward to review. DVC keeps Git focused on source, metadata and reviewable changes, while content-addressed artifacts live in storage designed for large files.

Data Versioning & ML Reproducibility

10.03 · Walkthrough

DVC for Data and Pipelines

Track a dataset and preprocessing pipeline with DVC and reproduce the same training dataset from versioned artifacts.

No video curated for this lesson yet

This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.

The written notes below cover this idea in full - you lose nothing by reading instead of watching.

DVC lets Git identify the exact data and preprocessing outputs behind an ML run without storing large artifacts in the repository. Code, metadata and pipeline definitions stay in Git, while datasets and generated files live in remote storage, giving teams a repeatable path from raw inputs to train-ready data.

What this lesson answers

how does DVC version datasets with Git
how does dvc repro decide what to rerun
how to reproduce an ML training dataset

Notes

DVC brings version-control discipline to data and pipeline artifacts without forcing large files into Git. A working pattern is to keep source code, DVC metadata, and pipeline definitions in Git, while the actual datasets, model files, and intermediate artifacts live in a DVC remote such as S3, GCS, Azure Blob, or shared storage. When you run dvc add on a raw dataset, DVC computes a content hash and writes a small pointer file that Git can track. This means a Git commit can now identify not only the exact code version, but also the exact dataset version used by that code.

Common questions

What does DVC store in Git?: DVC stores small metadata files, pipeline definitions and lock information in Git. The large files themselves, such as raw datasets, cleaned datasets, model outputs and intermediate artifacts, are kept in a DVC remote. A Git commit can then point to both the code and the exact data versions used with it.
How does DVC make preprocessing reproducible?: DVC records the inputs, commands, parameters and outputs for each pipeline stage. When reproduction runs, it compares the recorded state with the current project state. Unchanged stages can reuse existing artifacts, while changed dependencies or parameters cause only the affected work to run again.
Why not just commit processed datasets to Git?: Git is poor at handling large changing data files, and committing generated datasets directly makes repositories heavy and awkward to review. DVC keeps Git focused on source, metadata and reviewable changes, while content-addressed artifacts live in storage designed for large files.

Short definition: what is DVC for Data and Pipelines?

10.02 · Data Versioning & ML ReproducibilityDataset Versioning Principles 10.04 · Data Versioning & ML ReproducibilityLakeFS for Data Lake Branches