5.1 Reproducibility

🖥️ Lecture slides — Session 27 (Fri Dec 4)

Reproducibility is the entry ticket for trust in computational science. If your result cannot be regenerated from your data and code, it is an anecdote, however good the plot looks.

Definitions: reproducibility vs. replicability¶

Definitions vary across fields. We adopt the ones from the 2019 National Academies report Reproducibility and Replicability in Science Engineering et al., 2019:

Reproducibility is obtaining consistent results using the same input data, computational steps, methods, code, and conditions of analysis. This is often called computational reproducibility. It involves the original data and the original code.
Replicability is obtaining consistent results across new studies aimed at the same scientific question, each collecting its own data. Two studies replicate if their results agree within the uncertainty inherent in the system.

Two related terms from the same report: verification asks whether the code is correct, and validation asks whether the model predicts data it was not built on. Note the trap: a bug reproduces perfectly. Exact reproducibility does not certify correctness — it certifies that the pipeline is deterministic and complete. Validation on held-out data (the Chapter 3 discipline of hidden test sets) is what catches a reproducible mistake.

For a reproducible analysis, the report asks scientists to provide three things:

Data: the input data, or scripts that download or generate it, plus intermediate products where regeneration is expensive.
Methods: the computational steps and parameters, in executable form where possible.
Computational environment: operating system, hardware, and library dependencies.

Reproducibility in machine learning is usually accepted within a stated tolerance rather than bit-for-bit, because training involves randomness and floating-point non-determinism. Deciding what tolerance is acceptable is part of your job as the scientist; we return to it below.

Why the stakes rose when agents started writing code¶

Until recently, the main threat to reproducibility was human sloppiness: an untracked notebook cell run out of order, a CSV edited by hand, a “final_v3_REAL.ipynb”. Those threats still exist. The new one is scale. An agent can generate a complete analysis — data download, feature engineering, model, figures, prose — in minutes. The output is fluent and internally consistent. It can also be wrong in ways that fluent output hides: a leaked feature, a filtered dataset that silently dropped the hard cases, a metric computed on the training split.

You cannot review agent-generated code at the speed agents generate it. What you can do is make the checks executable. A pinned environment means the agent’s code runs against the same libraries tomorrow. Scripted transforms mean every step from raw data to figure is inspectable and rerunnable. CI means every change is re-executed before it lands. This is the theme of the whole chapter: when you delegate the typing, you must not delegate the verification. The course AI policy (Chapter 1.8) says you must be able to defend every line you submit; the practices below are what make that defense possible for code you did not write by hand.

The reproducibility stack¶

Five layers, from the ground up.

1. Environment pinning¶

“Works on my machine” usually means “works with the library versions on my machine.” Solvers, defaults, and even random-number streams change between releases. The fix is a lock: a machine-readable list of exact package versions that a tool can recreate anywhere.

Three common tools:

pixi (used by this book): a pixi.toml declares dependencies; pixi.lock pins every package, for every platform. pixi install recreates the environment exactly.
conda-lock: same idea for conda environments — render an environment.yml into a lock file per platform.
uv: fast pip-compatible tool; uv lock pins a pure-Python project.

This book is its own worked example. The repository root has a pixi.toml that declares the toolchain:

[dependencies]
python = "3.12.*"
numpy = ">=2.0"
pandas = ">=2.2"
scikit-learn = ">=1.6"
pytorch = ">=2.4"
obspy = ">=1.4"
# ...

[pypi-dependencies]
mlgeo-synth = { path = ".", editable = true }

The declared constraints are loose (>=), but the committed pixi.lock resolves them to exact versions. Every student, every CI runner, and every agent working in this repository executes the same stack. When we upgrade a library, the lock file changes in a commit — the environment has a history, like the code.

For your project: commit the manifest and the lock file. An environment.yml without versions is a suggestion, not an environment.

2. Seeds and determinism¶

Randomness enters ML in many places: data shuffling, train/test splits, weight initialization, dropout (the random silencing of units during training, lesson 4.2), stochastic optimizers. Control it explicitly:

import numpy as np
import torch

rng = np.random.default_rng(42)   # NumPy: pass rng around, do not use global state
torch.manual_seed(42)             # PyTorch: CPU and GPU generators

Prefer np.random.default_rng(seed) over the legacy np.random.seed(): a Generator object is local, so two parts of your code cannot silently share and perturb one global stream. In scikit-learn, pass random_state= to every estimator and splitter that accepts it.

Know the limits. On GPUs, some operations use non-deterministic algorithms for speed (atomics in reductions, cuDNN autotuning). torch.use_deterministic_algorithms(True) forces deterministic kernels where they exist, at a performance cost, and errors where they do not. Parallel reductions can also differ across hardware because floating-point addition is not associative. The practical stance: make everything deterministic that can be cheap to make deterministic, then measure the remaining run-to-run variance and report results with that spread. A claimed improvement smaller than your seed-to-seed variance is not a result. The exercise in 5.2 quantifies exactly this.

3. Raw data is immutable; every transform is scripted¶

The raw data directory is read-only. Nobody — not you, not an agent — edits a raw file. Every change of form (cleaning, resampling, feature extraction, labeling) is a script that reads from data/raw/ and writes to data/processed/. If the processed data is ever in doubt, you delete it and rerun the scripts. This is the ai-ready-data discipline from Chapter 2 restated as a filesystem rule, and it is the single cheapest reproducibility practice available. Corollary: if a step happened only in a notebook cell you have since overwritten, it did not happen.

4. Containers, in one paragraph¶

Lock files pin your packages, but not the operating system, system libraries, or compilers underneath them. Containers (Docker, Apptainer/Singularity on HPC) freeze that whole layer into an image you can run anywhere. For most course projects a lock file is enough; reach for a container when you must run on a cluster you do not control, ship a service, or archive a result for the long term (journals and archives increasingly accept an image alongside the code). Same principle, one level lower in the stack.

5. Executable checks: this book’s CI¶

The strongest reproducibility claim is one a machine verifies on every change. This book’s own build is the case study: a GitHub Actions workflow (.github/workflows/build.yaml) runs on every pull request, installs the pixi-locked environment, and executes every notebook in the book during the MyST build (myst build --execute --html). If a code cell errors — because a library changed, a dataset moved, or an edit broke an earlier cell — the build fails and the pull request cannot merge. A link checker runs in the same job.

The consequence is a guarantee the prose alone could never make: every result you see in this book was computed, from scratch, in a clean environment, at the commit you are reading. Section 5.3 shows how to copy this pattern into your own project repository, and the class leaderboard (Chapter 3.5) turns the same idea into scoring: your submission is whatever CI can execute, nothing more.

Checklist¶

Before you claim a result is reproducible, check:

git clone + one documented command recreates the environment (lock file committed)
all randomness is seeded, and run-to-run variance under different seeds is measured
raw data is read-only; every transform from raw to figure is a script under version control
a fresh clone on another machine (or CI) reruns the pipeline end to end
the reported numbers match the rerun within your stated tolerance