Your reproducible workbench

Session 2 · Fri Oct 2 · Book sections 1.1–1.5, 1.9

🧰

Learning objectives and class products

By the end of the session, you should be able to evaluate whether a published computational result can be rerun, identify the computing resources a workflow requires, and distinguish an environment from a sandbox or container.

The class will produce a first reproducibility rubric. Each student will begin a repository that can supply evidence for that rubric.

Readings: 1.1–1.5 · Lab: 1.9 workbench setup

Open by naming the unfinished work from Wednesday. Today does not start a new topic. The class finishes the paper sprint, extracts evidence from it, and uses that evidence to write a rubric. The hardware and workbench material then answers one rubric question: what would another person need to rerun this result on another machine? The session produces two reusable artifacts rather than a page of notes: rubric version 0.1 and each student’s course repository. Suggested 80-minute allocation: framing 2 minutes; evidence and literature 8; paper sprint 12; matrix, strong result, and rubric construction 16; computing concepts 14; compute-choice activity 7; workbench lab 16; exit ticket 5. If time runs short, compress the CPU/GPU and HPC slides to their takeaways; they are covered in 1.4.

Measured reproducibility of published research

Study	Tested	Outcome
Begley & Ellis 2012	53 landmark cancer studies	6 confirmed
Open Science Collab. 2015	100 psychology experiments	36% replicated (97% originally)
Stodden et al. 2018	204 computational Science papers	26% reproduced
Stagge et al. 2019	360 hydrology articles	1.6% reproduced
Konkol et al. 2019	R code of 41 geoscience papers	2 ran cleanly; 46 of 97 figures differ
Massonnet et al. 2020	one Earth system model, two HPC systems	different climates

Baker 2016 survey, recomputed from the raw data: 64% of 95 Earth and environmental scientists had failed to reproduce another group’s result.

Where reproducibility has been measured, most published computational results could not be regenerated. The geoscience studies report rates comparable to, or lower than, those in other fields.

Two minutes. Read the bottom two rows aloud: they are our field. Stagge et al. sampled 360 of 1,989 articles published in 2017 in six hydrology and water-resources journals and could regenerate results for 1.6% of those tested; their 95% confidence interval for all 1,989 is 0.6 to 6.8%. The main bottleneck was not secrecy: 44% had only some artifacts, and most had no directions to run them. Massonnet et al. (Geoscientific Model Development, 2020) ran the older EC-Earth3 version on two HPC systems and obtained statistically distinguishable climates, traced to the treatment of Antarctic river runoff; the newer version passed. Konkol, Kray and Pfeiffer (IJGIS, 2019) reran, in a clean rocker/geospatial container, the R code of 41 open-access papers: 31 Copernicus geoscience articles from 2016–2017 plus the 10 most-cited Journal of Statistical Software R-package papers. 2 ran without issues, 33 ran after fixes, 2 ran partially, and 4 could not be run; 15 required contacting the authors (11 helped, 3 never replied). They logged 173 issues in 39 papers, the most common being a missing library (49 cases) and hard-coded directories (34). Of 97 figures regenerated from 28 papers, 78 had a different aspect ratio (cosmetic) and 46 differed in content: different curves, missing or different key numbers. Massonnet’s conclusion is worth quoting: the default assumption should be that Earth system models are not replicable under a change of HPC environment until shown otherwise. The Baker number is not in the article itself; it is recomputed from the raw survey file Nature posted on figshare (doi:10.6084/m9.figshare.3394951). Earth and environmental scientists were 95 of 1,576 respondents; 61 had failed to reproduce someone else’s result, 39 their own. Ask: which row would your sprint paper fall into?

This lecture in the literature

✗ Hydrology: results reproduced for 1.6% of 360 sampled articles.

Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, N. A., & James, R. (2019). Assessing data availability and research reproducibility in hydrology and water resources. Scientific Data, 6, 190030.

✗ Geophysics: 86% of 200 articles state data availability; 54% provide the data.

Ireland, M., Algarabel, G., Steventon, M., & Munafò, M. (2023). How reproducible and reliable is geophysical research? Seismica, 2(1). doi:10.26443/seismica.v2i1.278

✗ Aftershocks: a two-parameter model matches the AUC of a 13,451-parameter network.

Mignan, A., & Broccardo, M. (2019). One neuron versus deep learning in aftershock prediction. Nature, 574, E1–E3. On DeVries et al. (2018), Nature, 560, 632–634.

✓ Temperature reconstructions: independent methods and new proxies recover the same shape.

Wahl, E. R., & Ammann, C. M. (2007). Climatic Change, 85, 33–69 · PAGES 2k Consortium (2019). Nature Geoscience, 12, 643–649.

A rerun did not establish whether any of these results was correct; it allowed other researchers to examine and test them.

Six minutes, four stories. Stagge et al. is the hydrology study from the previous slide. Ireland et al. (Seismica, 2023) is our closest field: 200 geophysics articles and 20 journal policies; about 86% had a data availability statement but the original data were reachable for 54% and the software was named for 49%. A data availability statement does not guarantee access to the data. DeVries et al. (Nature, 2018) trained a 13,451-parameter network on more than 131,000 mainshock–aftershock pairs and reported AUC 0.849 against 0.583 for Coulomb stress change. Mignan and Broccardo (Nature, 2019) matched it with logistic regression on two parameters, AUC 0.85. A data scientist, Rajiv Shah, also argued that the same earthquakes appeared in training and test sets; the authors disputed it and Nature did not act. The number reproduced; the interpretation that deep learning found physics a simple model misses did not survive a baseline. The hockey stick is the success story: Mann, Bradley and Hughes (1998, 1999); McIntyre and McKitrick (GRL, 2005) challenged the principal-component centering; Wahl and Ammann (Climatic Change, 2007) reimplemented the method and found the result robust to those criticisms; PAGES 2k (Nature Geoscience, 2013 and 2019) used 692 new proxy records and seven methods and reached the same conclusion. That walk across the table is on the next slides. If time allows, add Resplandy et al. (Nature, 2018): an outside reader, Nicholas Lewis, found that systematic errors had been treated as random; uncertainty grew about fourfold and the paper was retracted in 2019.

Paper sprint completion

12 minutes in pairs

1. Confirm the paper, DOI, and scientific claim
2. Locate the data and its access conditions
3. Locate code, license, and dependency record
4. Identify the exact figure or table you would rerun
5. Record the first point where the evidence trail breaks

Then complete two sentences:

Another team could rerun this result because the authors provide ___

Another team would get stuck at ___ because ___

Then propose one observable criterion for the course rubric.

“Uses GitHub” describes a platform. “A fresh clone regenerates Figure 3 from documented commands” states an observable test.

Resume the paper-to-project activity without repeating the introduction. Each pair should finish one evidence trail and turn it into a testable criterion. Ask them to write literal locations when possible: repository URL, archive DOI, environment file name, figure number, and evaluation split. “Code unavailable” is useful. “Not reproducible” is too vague. Each proposed criterion must describe evidence an evaluator can inspect. Collect one strong trail and one broken trail, then sort suggestions aloud into broad dimensions. The rubric slide supplies a candidate structure only after students have generated evidence themselves and seen the four-cell table.

Same or different data, same or different analysis

	Same data	Different data
Same analysis	Reproducible	Replicable
Different analysis	Robust	Generalisable

Hold the research question fixed. The Turing Way definitions (book 1.1). The National Academies (2019) use the top row; some fields swap the two words. State the operational test, not the label.

Each cell corresponds to a distinct claim. Rerunning a notebook on the original data tests only reproducibility, the upper-left cell.

This is the table from Wednesday’s slide, now with all four cells. Read each cell in our terms, using GNSS displacement as the running example. Reproducible: same NGL time series, same notebook, fresh clone, same trend. Robust: same series, but a different seed, split design, preprocessing choice, or model family, compared with a simple baseline. Replicable: the same pipeline on other stations, another region, or a later period. Generalisable: independent groups with different data and methods reach the same conclusion. The Turing Way gives its robust example as one pipeline in R and another in Python. The National Academies report used in Chapter 5.1 defines reproducibility with the same data, code and conditions, and replicability with each study obtaining its own data, which matches the top row. Older computational-science usage swaps the words; that is why the rubric asks for the test, not the label. Ask: which cell did your sprint paper claim, and which cell did its evidence support?

What each test establishes

Passed	Rules out	Still possible
Reproducible	missing steps, unbuildable environment	a bug, a leak, a bad metric
Robust	dependence on one seed, split, or choice; no baseline	a quirk of this data set
Replicable	dependence on one station, region, period	a shared blind spot in the method
Generalisable	dependence on one data set and pipeline	nothing is settled for good

A conclusion gains support as it persists across these tests. A complete report also states the conditions under which the result is expected to fail.

Point out that reproducibility is necessary and not sufficient; the Turing Way lists “being reproducible does not mean the answer is right” among its barriers to reproducible research. A bug reproduces perfectly. Map the two cases: the aftershock network passed reproducible and failed robust once a baseline was added. The hockey stick passed all four, over two decades, by other groups. Then map the course: homework is graded on reproducible; hidden-test leaderboards in Chapter 3 test a form of replicable; the final project should report robustness (seeds, split designs, baseline) and one replication on held-out locations or periods. Ask students to add one robustness criterion to the rubric on the next slide.

Class rubric v0.1

Dimension	Observable evidence
Scientific claim	one result and its success condition are stated precisely
Data and provenance	source, version, license, and processing history are recorded
Code and workflow	a fresh clone exposes the full path from data to result
Environment and compute	dependencies, hardware needs, and run commands are specified
Evaluation	split, baseline, metric, and uncertainty match the claim
Publication and credit	outputs are citable; people, data, software, and AI help receive credit

0 missing · 1 present but incomplete · 2 another team can verify it

Rubric criteria should describe inspectable evidence and the scientific claim that the evidence supports.

Compare the class suggestions with this candidate structure. Merge or rename dimensions rather than declaring this table final. The scale defined here (0 missing, 1 present but incomplete, 2 another team can verify it) stays fixed for every later rubric version, including Monday’s AI rows. The course rubric begins with computational reproducibility and honest evaluation. Save the class’s revised wording for rubric version 0.2 on Monday.

Rubric calibration

Before using the rubric for grades, test whether two evaluators interpret it similarly.

1. Select one paper or repository
2. Score each dimension independently
3. Compare scores and identify the source of disagreement
4. Revise the criterion or add an example

Disagreement may reveal ambiguous wording, missing evidence, or a substantive difference in scientific judgment.

Introduce inter-rater agreement as a design problem rather than a statistic to calculate today. A rubric can look precise while allowing different interpretations of “documented,” “available,” or “verified.” Ask two pairs to score the same evidence and explain any difference. The class should revise criteria that produce disagreement because the expected evidence is unclear. Preserve disagreements that reflect legitimate scientific judgment, such as whether an evaluation metric matches the intended use.

Hardware resources

Hardware is the physical machinery that executes and stores the work.

Resource	What it does	Typical bottleneck
CPU	general instructions; many scientific tasks	serial code or too few cores
GPU	many similar operations in parallel	model/data transfer and GPU memory
RAM	working memory for running processes	arrays or tables do not fit
Storage	persistent files and checkpoints	capacity and read/write speed
Network	moves data between systems	bandwidth, latency, access rules

A computational requirement should identify the relevant resource, its scale, and the evidence used to estimate it.

Use one concrete comparison. A pandas join on a moderate table usually needs CPU and RAM; a convolutional neural network often benefits from a GPU; neither helps if the data cannot move from storage quickly enough. A GPU is not a generally faster CPU. It accelerates workloads that expose large amounts of parallel work and use supported software. Ask students which resource limited the largest computation they have already attempted.

CPU and GPU execution models

CPU	GPU
a small number of sophisticated cores	many simpler arithmetic units
strong performance for branching and sequential logic	strong performance for uniform parallel operations
usually accesses large system memory	usually uses separate, capacity-limited device memory
common target for preprocessing and general scientific code	common target for tensor operations and neural-network training

Moving data between CPU memory and GPU memory has a cost. Small models, irregular algorithms, and slow data pipelines may gain little from a GPU.

Avoid equating GPU use with machine learning sophistication. The speedup depends on arithmetic intensity, parallelism, memory capacity, and software support. A GPU can remain idle while a CPU data loader or network filesystem limits throughput. Conversely, matrix and convolution operations can use thousands of parallel arithmetic units efficiently. Ask students to identify the parts of a typical training pipeline that remain on the CPU and the point where data transfer occurs.

Where computation runs

Context	What it means	Good fit	Main tradeoff
Local laptop	hardware you control directly	exploration, small data, offline work	limited memory, storage, and uptime
Back-end Linux server	remote shared machine reached through a network	longer jobs, shared data, stable services	permissions and shared resources
Cloud server	rented remote compute created on demand	temporary scale, public services, special hardware	cost and data governance
HPC system	many managed compute nodes plus a scheduler	large parallel jobs, many experiments	queues, modules, storage policy

A computing center operates shared hardware, storage, networking, schedulers, and support. Hyak is UW’s HPC system.

Demystify remote computing. A server is a computer that provides resources over a network. A back-end Linux server may be one powerful machine or a service entry point. A cloud server is still physical hardware in a data center, but the user rents a virtual machine or managed service. An HPC system combines many nodes and uses a scheduler so users request resources rather than taking them directly. A computing center is the organization and facility that operates these systems. Ask students which context they have used, even indirectly through a web service.

How an HPC job reaches hardware

Stage	What happens
Login	the user connects to a login node and prepares code, data references, and a job request
Resource request	the job specifies cores, memory, GPUs, wall time, and sometimes node type
Scheduling	the scheduler waits until a compatible allocation is available
Execution	the job runs on one or more compute nodes, usually without an interactive desktop
Persistence	logs, checkpoints, and results must be written to approved storage before the allocation ends

Requesting substantially more resources can increase queue time and allocation cost. Requesting too little can terminate the job.

Clarify the distinction between login nodes and compute nodes. Students should not run heavy jobs on a login node. The scheduler protects shared resources and records requests; it is not merely an inconvenience. Wall time, memory, and GPU requests become part of computational provenance because they can affect whether a workflow completes. Array jobs suit many independent files or parameter combinations, while distributed jobs require software that coordinates work across nodes.

Software boundaries

Term	Question it answers
Operating system	Which processes, files, devices, and users does the machine manage?
Computing environment	Which interpreter, packages, libraries, and exact versions run the code?
Sandbox	Which files, network services, tools, and actions may this process access?
Container	Which operating-system-level dependencies travel with the application?

A lockfile fixes package versions, a sandbox limits what a process may access, and a container also carries operating-system dependencies.

Students often merge these words. The hardware can stay fixed while the environment changes. The same environment can run inside different sandboxes. A sandbox may block the network or restrict writes even when every required package is installed. A container packages a broader software stack than a Python environment but still runs on host hardware and a host kernel. Pixi manages the course environment. The Codex workspace and many notebook services also impose sandbox boundaries. Connect this vocabulary to error diagnosis: “package missing” differs from “permission denied” and from “out of GPU memory.”

Minimum computational provenance

A reproducible run should preserve enough information to reconstruct both software and execution context:

• source revision and uncommitted changes, if any
• data identifiers, versions, and preprocessing parameters
• environment specification and lockfile
• command, configuration, random seed, and requested resources
• logs, warnings, output checksums, and the result selected for reporting

A published figure does not record the software, data, and parameters that produced it, so that information has to be preserved alongside the figure.

Different projects need different levels of capture. A small notebook may need a repository revision, environment lockfile, data DOI, and executed notebook. An HPC training run may also need scheduler directives, GPU type, random seed, checkpoint, and logs. Hardware identity matters when numerical libraries, nondeterministic GPU operations, or memory constraints can change behavior. The rubric should ask for material provenance rather than indiscriminate logging.

Compute choice activity

Choose a location, the likely limiting resource, and the rubric evidence you would record.

Case	Decision prompt
4 GB of GNSS data and exploratory plots	laptop, server, cloud, or HPC?
2 TB of continuous waveforms searched overnight	where should data and computation meet?
CNN training on 100,000 satellite tiles	when does a GPU help?
restricted collaborator data	what should the sandbox prevent?

Seven minutes in groups. State the assumptions behind one choice. Another group identifies a condition under which that choice would fail.

There is no single correct platform. Grade the reasoning. The 4 GB case often fits a laptop if RAM and storage are adequate. The waveform search benefits from moving compute near the archive and may suit a server or HPC array job. CNN training often benefits from a GPU, but data loading can dominate. Restricted data may require an approved server or cloud enclave with network and export controls. Each answer must include reproducibility evidence: machine class, requested CPU or GPU, memory, environment lockfile, command, and any access restrictions.

Workbench lab

Open 1.9 and produce evidence for the rubric:

1. Start pixi install for the book environment
2. Create MLGEO2026_UWNETID and add pixi.toml plus pixi.lock
3. Record the tested operating system and machine type in the README
4. Add one smoke test and name its pixi run command in the README
5. Review the diff, commit with a real message, and push

The repository should specify a runnable target, the environment that supports it, and an observable success condition.

Allow about 18 minutes. Start the large install first, then complete git and GitHub steps while packages download. Unfinished steps continue as HW1 through 1.9. Watch for one known failure: the book’s pixi.toml installs the book itself (mlgeo-synth, path .), so a copied file fails in a student repository until pixi remove --pypi mlgeo-synth is run; 1.9 Step 6 gives the command. The smoke test matters on Monday: the peer rerun runs the command named in each README, so a repository without one gives the peer nothing to execute. Circulate with rubric language: ask students to show evidence, not say that setup “worked.” Common failures separate cleanly using today’s vocabulary: command not found is a shell or PATH problem; dependency resolution is an environment problem; permission denied may be a sandbox or filesystem problem; killed processes often indicate memory limits. Students who finish should exchange repository URLs and attempt a fresh-clone check.

Exit ticket: input for rubric v0.2

Submit three lines before leaving:

• one rubric criterion you would keep
• one criterion you would sharpen, with observable evidence
• one computing choice your project may require

HW1 due Mon Oct 12 · Ch 1 quiz opens Tue Oct 6 · Monday begins by scoring your paper and workbench with the revised rubric

Reserve the final five minutes. Collect responses in Canvas or the Gaia workspace. Use them to prepare Monday’s rubric version 0.2. The continuity matters: Monday should begin with their words and their evidence rather than a new instructor-generated checklist. If several students still lack a functioning environment, direct them to the install clinic and allow the paper review and rubric work to count as today’s completed intellectual product.