Reproducibility

Definition

A result is reproducible when someone else, using the same data and the same code, can obtain the same result; it is replicable when new data collected by the same method give a consistent one. Reproducibility requires the data version, code, settings and environment to be recorded.

In plain terms

The US National Academies use these two words in this sense. Reproducibility is the more basic: if you cannot get the same number again from the same inputs, there is nothing to replicate.

In practice, “same inputs” means pinning everything that can change: the exact data as it stood at a point in time (Point-in-Time, or a content hash of the files), the version of the code, the parameters and random seeds, and the software environment. Keeping a log of each run, with these items, lets anyone, including you in six months, run it again.

Example

Two analysts run “the same” analysis and get different numbers.

A scored 0.78, B 0.75. After pinning the data snapshot, the library versions, the seed and the filter, both get 0.78.

Common misunderstandings

• “Sharing the code is enough.” Without the data version and environment, the same code can give a different answer.
• “Reproducible means correct.” It means repeatable; a wrong method is reproducibly wrong.
• “Fixing a random seed is cheating.” It makes a run repeatable; robustness to the seed should still be checked separately.