6.3 Build an Evaluation Set

This is the core graded exercise of the chapter. The rule it teaches:

Before you trust an agent with a task, build the evaluation set that measures it.

You already believe this rule for models — Chapter 3 made you evaluate every classifier against a held-out test set with ground truth, and the class leaderboard scored you on data you never saw. An agent is a model with tools. Same rule, same machinery: define the task precisely, assemble cases with known answers, score automatically, study the failures. The difference is that you now generate the ground truth, and this is exactly why physically motivated synthetic data has run through this whole course: mlgeo_synth gives you unlimited eval cases with free, exact truth.

We work one example end to end, then you repeat the procedure on a task from your own project domain.

The target agent task:

Given a raw daily GNSS displacement series, report the secular velocity in mm/yr and flag any coseismic offset.

A note on plumbing: the “agent” in this notebook is a Python function standing in for a real one, with realistic failure modes, so the notebook runs offline and in CI. The spec, the eval set, the scorer, and the loop are exactly what you would wrap around a live agent — but a live, sampled LLM agent adds three requirements the deterministic stand-in does not: validate its output schema before scoring (Step 3), run repeated trials per case and report pass rates (“Scoring without computable truth,” below), and track cost and latency. This notebook builds the first two; the third is bookkeeping.

🖥️ Lecture slides — Session 19 (Fri Nov 13)

Step 1: Write the task spec¶

Most “agent failures” are specification failures. A spec is not a wish (“analyze this GNSS data”); it is a contract precise enough that pass/fail is computable. Ours, version 1:

Note what is pinned down: input columns (so the agent cannot peek at truth columns), output schema (so scoring is mechanical), and numeric tolerances (so “pass” is not a mood). Note also what we did not think about yet — record length, noise level, offset size. The eval set is about to punish us for that, which is the point of building it.

Step 2: Generate the eval set¶

Twelve cases sweeping difficulty: clean baselines, strong seasonal signal, low signal-to-noise, offsets from large to tiny, short records, and a negative velocity so sign conventions get tested. Each case is a generator call plus its ground truth — which we know because we set it.

Look at C08 before moving on: a 4 mm offset under millimeter-level colored noise is invisible to the eye. Whether the agent should be required to find it is a genuine scientific question — and writing the eval set forced us to ask it, before any agent ran. Coverage design in one sentence: sweep every knob that changes the physics of the problem (signal size, noise, record length, confounding signals), include cases where the correct answer is “nothing there,” and include at least one case near the edge of physical detectability.

Step 3: Write the scorer¶

Mechanical, tolerance-based, no judgment calls at scoring time — all the judgment went into the spec. And one non-negotiable property: the scorer must survive malformed output. The most common failure of a live LLM agent is not a wrong number; it is output that violates the schema — a missing key, a number wrapped in prose, a string where a float belongs. A scorer that raises KeyError on such output cannot record the very failure it most needs to record. So score_case first validates the result against the spec’s output schema and converts any violation into a scored failure with a reason, never an exception. (The grading rubric at the end awards Scorer-soundness points for exactly this behavior — the worked exemplar now demonstrates what those points are for.)

Prove the guard works before trusting it — one deliberately malformed result, of the kind a live agent produces when it wraps the number in prose and forgets a key:

Step 4: Run it against the agent¶

Here is the simulated agent. Its strategy is the naive one a hasty (human or artificial) analyst produces: a straight-line fit for velocity, ignoring seasonal terms and offsets, and offset detection by scanning for the largest jump between 30-day means. Its failure modes are planted but realistic: velocity biased when an unmodeled signal is present, small offsets missed, noise excursions mistaken for offsets.

Step 5: Analyze failures, tighten, re-run¶

Three of twelve. Without the eval set, this agent would have shipped: its answers on any single series look reasonable. Read the failures by group — the table is the diagnosis:

• C02, C04, C06, C10 — velocity wrong whenever an offset is present. The straight-line fit absorbs the unmodeled step into the slope. On the 2-year record (C10) the bias reaches ~18 mm/yr. Correlated failure mode: the two outputs are not independent, so a missed model term corrupts a reported number.
• C07, C08 — small offsets missed. The hard-coded 12 mm threshold cannot see 8 or 4 mm steps. The spec never said what the minimum detectable offset should be; the eval set just made that gap visible.
• C05 — false alarm. Flicker noise wanders enough that some 30-day means differ by >12 mm with no earthquake at all. “No offset” cases earn their place in an eval set: a detector is only meaningful if it can stay silent.
• C09, C11 — velocity wrong on short records even with no offset. Two years of seasonal signal plus colored noise defeat a straight-line fit.

The failure analysis dictates the fix, and it is a spec fix as much as an agent fix. Spec v2 adds a required method clause — “estimate velocity by least squares with annual and semi-annual terms and, if an offset is flagged, a step function at the detected day” — the physical model from Chapter 2. The improved agent implements it:

Ten of twelve, and every physics failure is gone: offsets down to 4 mm are found (deciding on the fitted step amplitude beats thresholding raw jumps), the low-SNR false alarm disappears, velocities are right wherever the model is adequate.

The two survivors — C09 and C10 — are both 2-year records, and their velocity misses (~1.7–2 mm/yr) are not the agent’s fault. Velocity uncertainty from colored GNSS noise scales roughly as 1/T: two years of flicker noise cannot deliver ±1.5 mm/yr, no matter who analyzes it. Our spec demanded something the data cannot give. This is the second thing eval sets catch: unreasonable specs.

How far should the tolerance relax, then? Not to whatever makes the failures go away — that is grading on a curve you drew after seeing the exam. The tolerance is a property of the estimator under the physics, and we own the generator, so we can measure it: run the estimator on many fresh offset-free realizations at each record length and look at the distribution of its error.

The distribution answers the question. Ten-year records: 95th-percentile error near 0.3 mm/yr, so the original ±1.5 mm/yr is comfortable. Two-year records: the 95th percentile sits near 1.9 mm/yr — ±1.5 was never achievable, and a tolerance of ±3 mm/yr (about 1.5× the measured 95th percentile, headroom because 50 trials pin the tail only loosely and we sampled a single true velocity) is what the physics supports. Spec v3 writes that measurement into the contract. The general rule: derive tolerances from realizations of the estimator, not from a first guess — with a synthetic generator, the derivation costs five lines.

Twelve of twelve — and, more important, an audit trail: we know which capabilities were tested, which tolerances the physics supports, and what the agent could not do until its method changed. That table is what “I trust this agent with GNSS characterization” should mean.

This is the eval-driven loop:

1. spec → 2. eval set with ground truth → 3. scorer → 4. run → 5. failure analysis → tighten the spec or fix the agent → back to 4.

It is the same loop as model development in Chapter 3, with the eval set playing the role of the hidden test set. Two cautions transfer with it. First, an eval set you iterate against is a validation set; if the stakes are high, hold out fresh cases (new seeds!) for a final check, exactly as the leaderboard held out its test data. Second, the eval only covers what you put in it — our agent is validated for one-component series with at most one offset, and claiming more would be the overreach that Chapter 7.2 teaches you to write down honestly.

Scoring without computable truth¶

Everything above scored against exact numeric truth: abs(est - truth) <= tol settles every case. Many tasks you will actually delegate have no such number. The clearest example is the one this course is building toward: the pre-submission review agent you will construct from your own quality rubric, whose job is to critique a paper before you submit it. A review has no velocity_mm_yr. Its output is a judgment, and “is this a good review?” is itself a judgment.

The eval machinery survives, with three substitutions.

1. The scorer becomes a rubric. Decompose “is this review good” into binary checks a reader can settle without asking you: names the leakage flaw, every criticism cites a specific line, figure, or number, proposes a concrete fix for each flaw, does not praise a defect. You built exactly this kind of rubric in 6.2 part (c). And the planted-truth trick from this notebook still applies: feed the review agent papers with known, planted defects — a leakage bug, a fabricated citation, a tuned-on-test threshold — and “found the planted flaw” becomes a computable check again. Ground truth for a subjective task is something you engineer, not something you find.

2. You must measure whether the rubric can be applied consistently. A rubric only functions as a scorer if two raters applying it to the same output reach the same verdicts. That is measurable: have two raters score the same output (the partner swap in 6.2 part (c) produces exactly this data) and compute their agreement. Raw percent agreement is not enough — if most checks pass, two raters agree often by chance — so report Cohen’s kappa, agreement corrected for chance. It fits in five lines:

Report both numbers. The first pair agrees on 83% of checks and kappa is 0.63 — real but imperfect consistency. The lenient pair agrees on 92% of checks and kappa is 0.00: every bit of that agreement is what two yes-sayers produce by chance. Low kappa is a rubric defect before it is a rater defect: each disagreement marks a criterion whose wording two readers resolved differently. Rewrite that criterion, re-score, re-measure. A rubric two humans cannot apply consistently is not ready to be handed to an LLM judge.

When the second rater is a model (as in 6.2 part (c), and as your review agent’s own scoring will be), add the controls for the judge biases you measured there, applied at scoring time:

• Order. Score the pair as (A, B) and again as (B, A); a verdict that flips with position is position bias, not signal.
• Verbosity. Score each rubric check independently instead of asking for an overall impression — length inflates holistic scores far more than it inflates per-check answers.
• Blinding. Do not tell the judge which output came from your own agent, and strip any self-identifying preamble before scoring.

3. Single verdicts become pass rates. agent_v2 returns the same dict on every run; a sampled LLM does not. Run a live review agent five times on the same paper and it will catch a planted flaw in some runs and miss it in others. So the per-case unit of measurement changes from pass/fail to a pass rate with sampling uncertainty. Below are recorded results from a review agent run N=5 times per case on six short analyses with planted defects — the transcripts were recorded once and rubric-scored, so this cell runs offline and in CI like everything else:

Read P04: a single-run eval would have called it pass or fail on a coin flip — run once and ship, and your conclusion about the agent depends on the draw. And the standard errors say what a difference means: at N=5, a rate near 0.5 carries ±0.22, so one agent version scoring 3/5 where another scored 2/5 on a case is not evidence of improvement. Three consequences for your capstone eval: report rates, not verdicts; fix N in the spec before running; and demand a margin larger than the standard error before believing a change helped (or increase N until the error bars separate).

Taken together — rubric checks over planted defects, two-rater kappa on the rubric itself, order/verbosity/blinding controls, pass rates over N runs — this is the complete scoring machinery for the review agent you will build in the reading arc. It grades papers instead of velocities, and nothing about that exempts it from evaluation.

Optional: the same eval against a live model¶

Everything graded in this chapter runs offline: the simulated agent and the recorded transcripts are the graded baseline, and CI executes them without a network. If you have the hardware and the inclination, the same harness runs unchanged against a real open-weights model — OLMo 2, Ai2’s fully open model (weights, training data, and code all released) — served locally by Ollama:

ollama pull olmo2               # 7B default: ~4.5 GB disk; ~8 GB RAM to run; CPU works, slowly
export MLGEO_LIVE_EVAL=1        # opt in, then launch Jupyter from the same shell

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, so the “agent” below is one HTTP call. We hand the model 30-day means of the series (about 120 numbers for a 10-year record) — which is deliberately the bad-calculator setup from 6.1: a text engine given a regression problem. Expect schema violations and wrong velocities. That is the pedagogical point: the schema guard records the malformed outputs instead of crashing, the tolerances judge the numbers, and at nonzero sampling temperature you would need the repeated-trials machinery from the previous section. A capable live agent would instead write and run code on the raw series; wiring that up is a project-scale exercise, not a notebook cell.

The cell runs the live eval only when you have opted in with MLGEO_LIVE_EVAL=1 and a local Ollama server has an olmo2 model pulled; otherwise it prints a note and moves on. It cannot fail a build: no opt-in means no network call, and even an opted-in run that errors degrades to a message. Nothing in it is graded, and no assignment requires it.

Assignment: an eval set for your project domain¶

Repeat steps 1–5 for an agent task from your project’s domain. You choose the task; it must be one you would actually delegate. Examples to calibrate scope: “pick the P-wave arrival within ±0.1 s” (seismogram_dataset), “classify lithology from a geochemistry row and abstain when uncertain” (geochem_table), “estimate b-value from a magnitude catalog” (gutenberg_richter_magnitudes), “flag the rare events in this sensor series” (inject_rare_events).

Deliverables, in one notebook:

1. Task spec — inputs, output schema, tolerances, and any required method constraints. Written first, versioned if it changed. Tolerances derived from realizations where you own the generator, not guessed.
2. Eval set — at least 10 cases from an mlgeo_synth generator (or your own generator with documented ground truth), sweeping difficulty along at least three axes, including negative/“nothing there” cases and at least one case near the edge of detectability.
3. Scorer — mechanical, runs with one function call, and returns a scored failure (never an exception) on missing or malformed agent output, as in Step 3. Tolerance-based where truth is computable; if your task’s output is a judgment, use rubric checks over planted defects instead, and report two-rater agreement (percent + kappa) on at least a subset (“Scoring without computable truth”).
4. Run + failure analysis — run your (real or simulated) agent, group the failures, and diagnose each group: agent limitation, spec defect, or physical impossibility?
5. Writeup — half a page: what the eval set covers, what it deliberately does not, and what you would now trust the agent to do unsupervised.

Grading rubric (100 points):

Every rubric row is demonstrated in the worked exemplar above — the malformed-output handling that Scorer soundness asks for is Step 3’s validate_result, and the tolerance justification that Spec clarity asks for is the 50-realization measurement before spec v3.

One warning from experience: the most common way to lose points is an eval set your agent passes 10/10 on the first try. That is not a good agent; that is an easy exam.