Why are random train and test splits a problem for ML reproducibility?

If the split changes between runs, evaluation metrics are no longer measuring against the same records. A model may look better or worse because the validation or test data changed, not because the model, features or hyperparameters improved. Stable splits give you a fixed comparison point across experiments and rebuilds.

How do hash-based dataset splits stay stable as data grows?

Each record is assigned using a durable key, a chosen hash function and fixed threshold ranges for train, validation and test. When new records arrive, the same rule assigns only those new records. Existing records keep their previous split because their key and the split rule have not changed.

What key should be used for reproducible training splits?

Use a stable identifier for the real-world entity you need to keep together, such as a user, account or document. If several rows can belong to the same entity, split on the entity key rather than the row key. That prevents related examples appearing in both training and evaluation data.

Data Versioning & ML Reproducibility

10.07 · Walkthrough

Reproducible Training Splits

Create deterministic train, validation, and test splits that remain stable as new data arrives.

No video curated for this lesson yet

This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.

The written notes below cover this idea in full - you lose nothing by reading instead of watching.

Deterministic train, validation and test splits come from assigning records with a stable entity key and hash rule, not from row order or fresh randomness. Existing records keep their split when data is rebuilt or appended, so metric changes can be traced to data, features, code or hyperparameters rather than accidental reshuffling.

What this lesson answers

how to make train test split reproducible
stable validation split when new data arrives
avoid data leakage in deterministic dataset splits

Notes

Reproducible training splits are about making sure that every time you build a dataset, the same records land in train, validation, and test, even if the pipeline is rerun on a different machine or at a later date. This matters because model metrics are only meaningful when they are measured against a stable comparison point. If rows are randomly split each run, changes in performance may come from different data rather than from changes in features, model code, or hyperparameters.

Common questions

Why are random train and test splits a problem for ML reproducibility?: If the split changes between runs, evaluation metrics are no longer measuring against the same records. A model may look better or worse because the validation or test data changed, not because the model, features or hyperparameters improved. Stable splits give you a fixed comparison point across experiments and rebuilds.
How do hash-based dataset splits stay stable as data grows?: Each record is assigned using a durable key, a chosen hash function and fixed threshold ranges for train, validation and test. When new records arrive, the same rule assigns only those new records. Existing records keep their previous split because their key and the split rule have not changed.
What key should be used for reproducible training splits?: Use a stable identifier for the real-world entity you need to keep together, such as a user, account or document. If several rows can belong to the same entity, split on the entity key rather than the row key. That prevents related examples appearing in both training and evaluation data.

Short definition: what is Reproducible Training Splits?

10.06 · Data Versioning & ML ReproducibilityTraining and Serving Data Lineage 11.01 · Feature Engineering & StoresFeature Engineering Lifecycle