DVC for Data and Pipelines
DVC for data and pipelines is a way to version datasets, derived artifacts, and preprocessing steps alongside Git-tracked code, while storing large files outside Git. It records content hashes, dependencies, commands, parameters, and outputs so a team can recover or rebuild the exact training data used for an experiment.
The problem is that Git is good at source code but poor at large, changing datasets and generated files. In machine learning projects, the code commit alone rarely explains a result: raw data may have changed, preprocessing may have been rerun, parameters may have drifted, or a cleaned dataset may have been overwritten. DVC makes those non-code inputs part of the same reviewable history, instead of relying on filenames, shared folders, or memory.
DVC works by putting small metadata files in Git and the heavy artifacts in a separate remote store such as object storage or shared storage. When a dataset is added, DVC hashes its contents and records a pointer to that exact version. For pipelines, it records stages in a pipeline file: dependencies, commands, parameter files, and outputs. A lock file then pins the concrete artifact versions produced by those stages.
When reproduction is requested, DVC compares the recorded graph with the current workspace. If an input file, command, parameter, or upstream output has changed, the affected stage is rerun; otherwise the existing artifact can be reused. This is the mechanism that turns a vague claim like “trained on the cleaned data” into a precise chain from raw inputs through transformation code to the final training dataset.
The trade-off is extra discipline and infrastructure. Engineers must commit DVC metadata, push and pull artifacts from the remote, define pipeline stages accurately, and avoid side effects that DVC cannot see. It does not make messy data clean, guarantee deterministic code, or replace experiment tracking. Reproducibility still depends on what you declare as dependencies, how stable your commands are, and whether external services or random sampling are controlled.
Engineers meet DVC in repositories where data science work needs code-reviewable provenance. Typical touchpoints are commands that add raw data, pipeline definitions for preprocessing, parameter files for feature choices, lock files in pull requests, and CI jobs that fetch artifacts and rebuild derived datasets. The common misunderstanding is that DVC is just “Git for big files”; its more important role is preserving the relationship between data, code, configuration, and outputs.
Common questions
- Does DVC store datasets in Git?
- No. DVC stores small pointer and pipeline metadata in Git, while the large files live in a DVC remote. The pointer identifies artifact content by hash, so checking out a Git commit can identify the matching dataset version without bloating the Git repository.
- How is DVC different from naming files with dates or versions?
- Filenames are conventions; DVC records content and relationships. A renamed or overwritten file can look trustworthy while containing different bytes. DVC hashes artifacts and records which commands, inputs, parameters, and outputs belong together, so the project history can prove which data a pipeline used.
- Does dvc repro always rerun the whole pipeline?
- No. DVC checks the dependency graph and reruns only stages whose recorded inputs, parameters, commands, or upstream outputs have changed. If nothing relevant changed, it can reuse the existing output. The exact behaviour depends on whether the pipeline accurately declares everything the stage reads and writes.
- Can DVC guarantee identical model training results?
- Not by itself. DVC can reproduce the versioned data and pipeline artifacts, but training may still vary because of randomness, hardware differences, library versions, or undeclared external inputs. It is one part of reproducibility: it makes the data and preprocessing lineage explicit and recoverable.