6.5 The Reading Arc: From Papers to a Review Agent

🖥️ Lecture slides — Session 19 (Fri Nov 13)

This page is the contract for the quarter’s reading assignments. There are no reading reports in this course. Instead, four staged assignments run across the quarter and build on each other: you verify literature with an AI assistant, dissect what makes papers good or bad, write your own quality standards, and finish by building a pre-submission review agent from those standards and evaluating it with the machinery of 6.3. Each stage has a graded deliverable, listed in the table at the end. Alongside the stages, every student gives one short paper-pulse talk during the quarter, and every student in the room scores every talk through the peer-feedback survey — presenting and evaluating are both practiced skills here, not one-time performances. Week-by-week scheduling lives in the course schedule; this page defines what you submit and how it is graded.

The arc assesses outcomes 10 (evaluate AI agents as scientific instruments), 11 (integrate AI assistance while keeping intellectual ownership), and 12 (translate and appraise results for distinct audiences).

Stage 1 — AI-assisted literature review¶

The topic is the same for everyone, localized to your field: the opportunities and challenges of AI in your subfield of the geosciences. Two shared anchor readings frame it — one survey of machine learning across the geosciences and one critical paper on its limits (current picks are listed on the syllabus; the pairing changes as the literature moves). From that starting point, drive an AI assistant through a literature review localized to your own subfield: ask for the state of the field, the canonical papers, the open questions. Then verify everything.

This topic is not decoration. The papers you surface here are your raw material for the rest of the arc: your Stage 2 dissection targets, your paper-pulse talk, and the position you will revisit in Stage 4 after you have measured an agent yourself.

The deliverable is the verification log, not the summary. The assistant’s summary is raw material; your graded work is the audit of it. For every citation the assistant produced, the log records:

Close the log with a short paragraph: how many citations survived, which failure mode dominated, and what the verification cost you in time relative to the assistant’s drafting time. That ratio is the honest price of AI-assisted reading, and you will quote it back to yourself all quarter.

A summary with unverified citations, however fluent, scores zero. This is the same duty stated in the course AI policy, applied to reading instead of code.

Stage 2 — Anatomy of good scientific papers¶

You receive two exemplar papers in your general area: one strong, one flawed. (The flawed one is real or lightly edited; you are not told which is which.) Dissect both.

Deliverable: an annotated defects-and-strengths list, organized by paper section. For each section of each paper, entries that answer three questions:

• Methods defensibility. Could the authors defend each methodological choice if questioned — splits, baselines, tolerances, exclusions? Where is a choice unstated, unjustified, or leakage-prone?
• Figure honesty. Do the figures show what the captions claim? Axis ranges, error bars or their absence, cherry-picked examples versus representative ones.
• Claim–evidence linkage. For each claim in the abstract and conclusions, which figure, table, or number supports it? Claims with no traceable support get flagged; so does strong evidence the authors undersold.

Every entry cites a specific line, figure, or number. “The methods are weak” earns nothing; “Section 3.2 tunes the detection threshold on the same events used in Table 2” earns credit. This is the skill the review agent will need in Stage 4, practiced by hand first.

The paper-pulse talk (part of Stage 2’s grade)¶

Once during the quarter — sign-ups open in week 2 — you give a 4-minute talk plus one question at the top of a class session, on one paper from your Stage 1 literature review. The format is fixed:

1. The paper’s central claim, in one sentence a non-specialist follows.
2. The evidence behind it — which figure or number carries the claim.
3. One strength and one flaw, judged against the class’s rubric-in-progress.
4. One sentence of disclosure: what the AI assistant did in your reading workflow, and what you verified.

Two talks run per session from week 3 through week 10, so by rubric-writing time (Stage 3) the class has built a shared corpus of dissected papers — dozens of worked examples of what good and bad look like, presented by the people who found them. The talk is graded inside Stage 2’s points, on the same standard as the written list: specificity earns credit, vagueness does not.

Everyone in the room scores every talk. A short peer-feedback survey (on Canvas, the same instrument all quarter) collects five rubric-mirrored ratings and two required free-text lines — one specific strength, one specific improvement — during each talk. Feedback reaches the presenter anonymized and screened. Completing the surveys on days you attend counts toward participation; your ratings are never graded for “correctness.” The survey is itself course content: it is rubric-as-scorer with many raters, and the class’s own agreement statistics on those ratings return as a live dataset when 6.3 asks how much a single rater’s verdict is worth. You will present once and evaluate roughly forty times; the second skill is the one the arc is really training.

Stage 3 — Build your own standards¶

Stage 2 gave you a vocabulary of defects and strengths. Stage 3, run through class discussion, turns it into your own explicit standard.

Deliverable: a quality rubric for papers in your field. The rubric states:

• Criteria — each one a check a reader can settle without asking you, phrased so that two readers reach the same verdict. “Methods are rigorous” is not a criterion; “every learned model is compared against a stated domain baseline” is.
• Weights — which criteria matter most in your field, and why (one sentence each).
• Pass bars — what a paper must satisfy before you would send it to a journal or a stakeholder. Name the criteria that are hard requirements versus advisory.

Declare your genre. Per the final project’s two tracks, you write the rubric for one of two genres and state which at the top:

• Journal-paper rubric — standards for a peer-reviewed research paper in your field.
• Technical-report / stakeholder-deliverable rubric — standards for a report written for a named decision-maker: does it state the operational question in the stakeholder’s terms, quantify what happens when the model is wrong, separate findings from recommendations?

The two genres share the evidence standards from Stage 2 but differ in what “good” means; a rubric that ignores its genre’s audience fails outcome 12 regardless of how carefully it is weighted.

Write the criteria as binary checks. Stage 4 will use this rubric as a scorer, and only binary, independently checkable criteria survive that conversion — this is the rubric-as-scorer requirement from 6.3’s “Scoring without computable truth” section.

Stage 4 — Build and evaluate your review agent¶

The capstone. Turn your Stage 3 rubric into a pre-submission review agent — a specified agent whose job is to critique a paper or report against your standards before you would submit it — and then evaluate that agent the way 6.3 taught you to evaluate any agent. The agent is worth nothing ungraded; the evaluation is the assignment.

Deliverable: one notebook or report containing six parts, mirroring 6.3’s assignment structure:

1. Task spec. Inputs (a manuscript or report), output schema (which rubric checks, in what format, with what required specificity — e.g., every criticism cites a line, figure, or number), and constraints. Written before you run anything.
2. Eval set. At least six short papers or reports with known, planted strengths and flaws — a leakage bug, a tuned-on-test threshold, a fabricated citation, a dishonest axis — plus at least one negative case with nothing planted, so you can catch an agent that criticizes everything. Ground truth for a subjective task is something you engineer: “found the planted flaw” is a computable check even when “is this review good” is not.
3. Rubric-as-scorer. Your Stage 3 rubric, applied as binary checks to the agent’s reviews. Score each check independently rather than asking for an overall impression, and apply the judge-bias controls from 6.2: score pairs in both orders, and blind the scorer to which output came from which agent.
4. Two-rater agreement. Swap one case’s agent reviews with a partner; both of you score them with your rubric. Report percent agreement and Cohen’s kappa (6.3 shows the five-line computation and why percent alone misleads — two lenient raters agree constantly by chance). Low kappa is a rubric defect before it is a rater defect: rewrite the ambiguous criterion, re-score, report both rounds.
5. Run and failure analysis. Run the agent N=5 times per case and report pass rates with binomial standard errors, not single verdicts — a sampled model catches a planted flaw in some runs and misses it in others, and at N=5 a rate near 0.5 carries about ±0.22. Group the failures and diagnose each group: agent limitation, rubric defect, or a case that was genuinely ambiguous.
6. Closing reflection (half a page). Return to your Stage 1 topic — the opportunities and challenges of AI in your subfield — and answer it again, now citing your own measurements instead of the literature’s claims: what your agent caught reliably, what it missed, what the kappa and pass rates say about delegating judgment. Positions that changed since Stage 1 are worth stating; so are positions that survived contact with the data.

Graded baseline: the offline recorded-transcript path. As everywhere in this chapter, the graded work runs offline. The course provides recorded review transcripts — a reference review agent run N=5 per case on a planted-defect corpus — and your rubric, scorer, agreement analysis, and failure analysis run against those recordings, in CI, with no network. You still submit your own agent’s full specification (Part 1) built from your own rubric; the recordings stand in for the runs.

Optional: the live path. If you have the hardware, run your own spec against OLMo 2 served locally by Ollama, exactly as 6.3’s optional section does (ollama pull olmo2, opt in with MLGEO_LIVE_EVAL=1; about 8 GB RAM, CPU works). Score the fresh transcripts with the same machinery. Nothing about the live path is required and it earns no extra points; the offline path is the graded baseline so that the assignment works in restricted environments.

Grading¶

100 points across the arc. Each deliverable also names the AI tools used and for what, per the disclosure policy.

Peer-feedback surveys are participation credit, tracked by completion on days you attend; they carry no arc points and your ratings are never judged for accuracy.

469 / 569 differentiation¶

The arc runs at two levels, following the course delivery rule (Apply-level for 469, Create-level for 569 on outcome 10):

The kappa measurement and the failure analysis are required at both levels; they are the parts of the machinery that do not dilute.

Why the arc ends here¶

By Stage 4 you have written the standards, encoded them as checks, and measured — with agreement statistics and pass rates, not impressions — how well an agent applies them. The agent enforces a rubric you wrote, weighted for reasons you can state, tested against defects you can name. When someone asks whether the AI wrote your paper or you did, the arc is the course’s answer: the assistant drafts and critiques, but the standards it enforces are yours, and you can defend every one of them without it in the room.