What should be versioned in a dataset strategy?

Version the raw input, cleaned or transformed outputs, labels, feature definitions and values, and the records assigned to training, validation, and test splits. Also store lineage: which input version, code version, parameters, and environment produced each artefact. The aim is to reconstruct the model’s data path, not just keep old files.

Are random seeds enough for reproducible train and test splits?

No. A seed helps only if the input records, ordering, sampling code, and library behaviour are identical. A safer approach is to persist the actual record identifiers in each split, together with the dataset version and sampling rule. Then comparisons remain meaningful even when implementation details or input ordering change.

Should raw data ever be overwritten?

Usually, no. Raw ingested data should be treated as an immutable reference, with corrections captured as new snapshots, patches, or derived datasets. Overwriting destroys the ability to reproduce old experiments and audit source changes. There are exceptions for privacy deletion or legal requirements, but those should be explicit and recorded.

Dataset Versioning Principles

Dataset versioning principles are rules for making each data artefact in a machine learning pipeline uniquely identifiable, reproducible, and traceable. They cover raw inputs, derived datasets, labels, features, and train or evaluation splits so engineers can explain exactly which data produced a model and recreate it later.

Machine learning systems fail in ways that ordinary code versioning cannot explain. The model may be unchanged, while a source table was backfilled, a label policy shifted, a feature job was edited, or a random split changed. Without explicit dataset versions, metric changes become ambiguous. Engineers cannot tell whether a regression came from code, data, annotation, sampling, or the interaction between them.

A practical strategy treats data artefacts as named snapshots with lineage. Raw data is usually append-only or immutable after ingestion, with source, capture time, schema, and provenance recorded. Derived datasets point back to the raw snapshot plus transformation code, parameters, and execution environment. Labels record the annotation source or rule set. Features record both computation logic and input data so offline training can be compared with serving behaviour.

The trade-off is operational weight. Storing snapshots, metadata, record identifiers, and lineage takes space and discipline, and it can slow down ad hoc exploration. The honest answer to how much to version is: it depends on risk, regulatory pressure, team size, and how expensive a wrong model is. Commonly misunderstood: versioning only the final training file is not enough if labels, features, or splits can change independently.

Engineers meet these principles when designing data lakes, feature stores, experiment tracking, model training pipelines, and CI checks for ML. In practice, the version is often a content hash, table snapshot, partition, manifest, or pointer tracked alongside code. Training, validation, and test splits should be stored as artefacts containing exact record identifiers and sampling rules, not silently regenerated during each run.

Go deeper

Everything DataDataset Versioning PrinciplesDefine a versioning strategy for raw data, transformed data, labels, features, and training splits.Open the lesson

Common questions

What should be versioned in a dataset strategy?: Version the raw input, cleaned or transformed outputs, labels, feature definitions and values, and the records assigned to training, validation, and test splits. Also store lineage: which input version, code version, parameters, and environment produced each artefact. The aim is to reconstruct the model’s data path, not just keep old files.
Are random seeds enough for reproducible train and test splits?: No. A seed helps only if the input records, ordering, sampling code, and library behaviour are identical. A safer approach is to persist the actual record identifiers in each split, together with the dataset version and sampling rule. Then comparisons remain meaningful even when implementation details or input ordering change.
Should raw data ever be overwritten?: Usually, no. Raw ingested data should be treated as an immutable reference, with corrections captured as new snapshots, patches, or derived datasets. Overwriting destroys the ability to reproduce old experiments and audit source changes. There are exceptions for privacy deletion or legal requirements, but those should be explicit and recorded.

Every term