Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

🖥️ Lecture slides — Session 27 (Fri Dec 4)

A tracked experiment (Section 5.2) records which code produced a number. That is only half the provenance. The other half is which data and which model. This page covers both, then shows how to wire the checks into CI.

Versioning data

Why git fails for data

Git stores every version of every file and computes diffs line by line. That design breaks on data three ways. First, size: git copies content into every clone, so a 5 GB waveform archive makes every git clone a 5 GB download, and ten versions make it fifty. Second, diffs: binary formats (HDF5, Parquet, GeoTIFF, miniSEED) do not diff meaningfully, so git stores near-complete copies per version. Third, hosting limits: GitHub rejects files over 100 MB outright. Data needs a different mechanism; the good news is that the mechanism can still be anchored in git.

The tools, one paragraph each

git-lfs (Large File Storage) is the smallest step up: git stores a small pointer file, and the real content lives on an LFS server, fetched on checkout. It keeps the git workflow intact and works well for a handful of medium files (model weights, a reference dataset). It gets expensive and slow at tens of gigabytes, and quota on GitHub’s LFS is limited.

DVC (Data Version Control) is git-lfs generalized to datasets and pipelines. dvc add data/raw/catalog.parquet writes a tiny .dvc metafile (a checksum plus a size) that you commit to git; the content goes to a remote you choose — S3, GCS, a lab server, even Google Drive. dvc checkout materializes exactly the data version matching the current commit. DVC also describes pipelines (dvc.yaml): stages with declared inputs and outputs, re-executed only when their inputs change.

lakeFS brings git semantics to an object store: branches, commits, and merges over an entire S3-style bucket. You can create a branch of a petabyte data lake in milliseconds (it is copy-on-write), run an experiment against it, and merge or discard. It is infrastructure a lab or data center runs, not a per-project tool — worth knowing it exists when you join a group that operates at that scale.

The minimum viable practice

You do not need any of these tools to be safe. The floor, achievable with what you already have:

  1. Raw data is immutable (Section 5.1). One directory, write-once, never edited.
  2. Checksums are recorded. A SHA256SUMS file, or a checksum in your download script, committed to git. Now “the data” is a verifiable object, not a filename.
  3. Every processed dataset is (script + raw data + parameters), all under git. Regenerating beats storing. Where regeneration is slow, store the product and the recipe.

This course’s data repository works exactly this way: notebooks fetch files with pooch, which takes a URL plus a known_hash and refuses to proceed if the download does not match:

import pooch
fname = pooch.retrieve(
    url="https://github.com/UW-MLGEO/MLGeo-dataset/raw/main/data/catalog.csv",
    known_hash="sha256:6f1c1a3f...",
)

If the file upstream changes, the hash check fails loudly instead of your analysis changing silently. That one argument is a data-versioning system in miniature.

Versioning models

A trained model is a derived product, like a processed dataset: it is (code version + data version + configuration + randomness). Saving only the weights throws away the provenance. Save the bundle:

torch.save({
    "state_dict": model.state_dict(),
    "config": config,                    # architecture + training hyperparameters
    "data_version": "catalog v2.1, sha256:6f1c1a3f...",
    "code_version": "git 3f2a9c1",
    "metrics": {"val_f1": 0.83, "test_f1": None},   # test stays hidden until the end
    "seed": 42,
}, "models/detector_v1.2.0.pt")

Version models semantically, like software. Bump the major version when the interface changes (different inputs or outputs — consumers must adapt); the minor version when behavior changes but the interface does not (retrained on new data, new architecture, same task); the patch version for fixes that should not change behavior (export bug, metadata correction). “The model” in your report should always mean a specific version.

Model cards

A model card is a short, standard description of what a model is for and where it breaks — introduced by Mitchell et al. (2019, “Model Cards for Model Reporting,” FAT* '19), now expected on model hubs. Ten lines is enough for a course project:

# Model card: detector_v1.2.0
- **Task**: P-wave arrival detection on 100 Hz, 3-component seismograms
- **Architecture**: 1-D CNN, 120k parameters (config in models/detector_v1.2.0.pt)
- **Training data**: catalog v2.1 (sha256:6f1c1a3f...), 2015-2022, Pacific Northwest
- **Metrics**: recall 0.92 at 1 false alarm/day on the hidden test set
- **Known limits**: recall drops to 0.60 below SNR 3; untested outside the PNW;
  not evaluated on borehole instruments
- **Intended use**: research catalog building. Not for earthquake early warning.
- **Contact / license**: mlgeo-team-4, MIT

The “known limits” and “intended use” lines are the ones that matter. They are also the ones an agent cannot write for you, because they encode judgment about what you did not test. Chapter 7.2 builds on this when you write the downstream-impact statement for your final project.

CI for science: copying this book’s pattern

This book’s workflow (.github/workflows/build.yaml) does three things on every pull request:

  1. Recreates the pinned environment with prefix-dev/setup-pixi, using the committed pixi.lock (with caching, so it is fast after the first run).
  2. Executes every notebook via myst build --execute --html. Any cell that raises fails the build.
  3. Checks every link (myst build --check-links), so references to data, papers, and other pages do not rot silently.

To copy the pattern into a project repository, the whole workflow is about twenty lines:

name: ci
on: [pull_request, push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: prefix-dev/setup-pixi@v0.9.0
        with: { cache: true }
      - run: pixi run pytest tests -q
      - run: pixi run jupyter nbconvert --to notebook --execute notebooks/*.ipynb

Adapt the last two lines to your project: unit tests for your processing functions, execution of the notebooks that produce your figures. Keep notebook runtimes short — use a small data subset or a SMOKE_TEST environment variable for CI, and run the full study outside it. If your pipeline cannot finish in CI even in reduced form, that is worth knowing: it means no one, including you, can cheaply verify it.

One more pattern from this course: scoring as CI. The class leaderboard (Chapter 3.5 exercise) is a workflow that executes your submitted predictor against a hidden test set and posts the score. The general lesson transfers to research: whenever a number matters — a benchmark, a leaderboard entry, a headline metric — arrange for a machine, not its author, to compute it.

Checklist

Chapter 5 — Workflows, Reproducibility & Rigor
5.2 Experiment Tracking
Chapter 5 — Workflows, Reproducibility & Rigor
5.4 Compute Beyond the Laptop