10.01 · Concept
The Reproducibility Problem in ML
Identify the data, code, environment, configuration, randomness, and infrastructure variables required to reproduce an ML run.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Reproducing an ML run requires more than the training code: you need the exact data, source revision, runtime environment, configuration, random inputs, and infrastructure assumptions that shaped the result. Missing any of these can make two apparently identical runs diverge in metrics, model artefacts, or behaviour.
What this lesson answers
- what makes an ML run reproducible
- which variables affect ML experiment results
- why identical ML code gives different results
Notes
The Reproducibility Problem in ML: Identify the data, code, environment, configuration, randomness, and infrastructure variables required to reproduce an ML run.
Common questions
- Why is ML reproducibility harder than normal software reproducibility?
- ML output depends on inputs that are often outside the application code: datasets, preprocessing, training configuration, random initialisation, library versions, hardware, and scheduler behaviour. In normal software, the same code and environment usually define the result. In ML, the learned artefact is produced by a run, so the run context matters.
- What should be captured to reproduce an ML run?
- Capture the dataset version, preprocessing code, model code, dependency versions, runtime image or environment, training configuration, seeds and randomness controls, hardware type, distributed execution details, and produced artefacts. The goal is to describe both what was executed and the conditions under which it executed.
- Can setting a random seed make an ML run reproducible?
- A seed helps, but it is not sufficient on its own. Other sources of variation include data ordering, parallel execution, nondeterministic hardware operations, changing dependencies, and differences in infrastructure. Treat the seed as one recorded input among several, not as a complete reproducibility mechanism.
Short definition: what is Reproducibility Problem in ML?