Working with agents:
policy and mechanism

Session 3 · Mon Oct 5 · Book sections 1.8 + 6.1

🤖

Today’s question

A fast, tireless, well-read collaborator — with no stake in being right.

What are the working rules?

Reading: 1.8 AI in your workflow + 6.1 From language models to agents

Frame from 1.8’s own sentence: an assistant is a fast, tireless, well-read collaborator with no stake in being right. It will produce plausible code for a method it misunderstood, cite papers that do not exist, and agree with a wrong framing. Everything today follows from that one sentence. Two halves to the session: mechanism (what an agent is and where errors physically enter — 6.1), then policy (badges, disclosure, verification — 1.8). Knowing where errors enter is what makes checking for them possible; that ordering is deliberate.

This lecture in the literature

✗ Finite attention: recall sags in the middle of long contexts — an agent “reading” your repository is sampling it.

Liu, N. F., Lin, K., Hewitt, J., et al. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.

✗ Trained to please: preference-tuned assistants agree with your framing — including when your framing is wrong.

Sharma, M., Tong, M., Korbak, T., et al. (2023). Towards understanding sycophancy in language models. arXiv:2310.13548 (also published at ICLR 2024).

✗ AI grading AI tracks human raters — but carries position and verbosity biases you must measure first.

Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track.

Three measured failure modes — attention, agreement, judging. Not anecdotes: experiments you can rerun.

Two minutes each; all three are cited in book section 6.1 and all three are ✗ on purpose — today is the failure-mode lecture. Liu et al. measured recall as a function of position in the context window: material in the middle of a long context is recalled worse than material at the ends, so an agent that “read” a 40-file repository has sampled it and may answer confidently from the parts it attended to. Sharma et al. (Anthropic) showed preference-tuned models systematically defer to the user — change your stated opinion and the model’s answer follows; a model tuned to please raters tends to please you, including when you are wrong. Zheng et al. built the LLM-as-a-judge methodology: strong agreement with human raters, but position bias and verbosity bias — relevant in week 7 when students build agent evals, and a preview that “judge” is a role you must itself evaluate.

What an agent is

an LLM + a set of tools + a loop over observations

• The model proposes an action; the harness runs it; the result returns as context
• Reading files, running Python, querying an earthquake catalog — all just tools
• “Did seismicity rate change after the M6.4?” → query, write the test, run, look, revise

The arithmetic moved to numpy, which is reliable. What to query, which test, what the result means — still the model’s judgment.

Demystify: an agent is not a new kind of mind, it is a language model that can emit structured tool calls instead of prose, plus a loop that executes them and feeds results back. Walk the seismicity example from 6.1: the agent queries an FDSN catalog, writes a rate-change test, runs it, reads the numbers, revises — a genuine multi-step scientific task. Then the trust relocation, which is today’s central mechanism point: tool use fixes the arithmetic problem (numpy computes), but everything the model still decides — which data, which test, whether completeness magnitude was handled, whether an empty result means “no earthquakes” or “wrong request” — remains unreliable. Agent errors are specification and judgment errors, which is why Chapter 6 evaluates agents end to end against ground truth rather than reading their code line by line.

Why the model cannot do the math itself

0 numeric types inside a language model — digits are vocabulary

3,650 daily GNSS values pasted into a prompt: a regression handed to a text engine

13.847 may tokenize as 13 · . · 847. Arithmetic becomes pattern completion over digit strings — fluent, roughly plausible, silently wrong.

A well-built agent writes and runs the numpy code. It does not “read” your numbers.

The mechanism in one breath: text is split into tokens, numbers get fragmented arbitrarily, and the model has no numeric type — so asking it to compute is asking it to complete a familiar-looking digit pattern. Ten years of daily GNSS displacements pasted into a prompt is the canonical mistake: the model will answer, fluently, with a roughly plausible trend, and there is no error message. The fix is not a better prompt; it is giving the model a calculator — which is exactly what tool use is. This is today’s in-class exercise: they will ask their own assistant for the P395 east velocity and check whether it wrote code or “read” the numbers.

Where errors enter the loop

• Attention: long contexts sag in the middle — an agent “reading” 40 files is sampling them
• Agreement: tuned on human approval, it tends to ratify your framing, wrong framings included
• Judgment: does an empty catalog result mean “no earthquakes” — or “wrong request”?

Long visible reasoning makes errors more persuasive, not less common.

Map each bullet to its paper from the literature slide. Attention is Liu et al. — practical consequence: never assume the agent saw the whole repository; point it at the specific files that matter and spot-check what it claims to have read. Agreement is Sharma et al. — practical counter: ask for the case against your framing, or state the opposite hypothesis and see whether the model follows you there too; students will measure this in the 6.2 lab (Fri Oct 23). Judgment is 6.1’s empty-query example — the error class no amount of model scale removes, because it is a specification problem. Close with the reasoning-model caveat from 6.1: models that generate long internal deliberation genuinely help on multi-step problems, but a wrong answer after ten thousand tokens of deliberation is still wrong — and reads as more convincing.

Hallucination is not ordinary error

• An ordinary error comes from a traceable process — it usually looks wrong somehow
• Hallucination is fluent fabrication, optimized to look right
• Canonical case: the fabricated citation — real journal, well-formed DOI, no paper
• Subtler: an extraction table where a few values appear nowhere in the source

Plausibility is the one test hallucination is guaranteed to pass. Verify against the data, never against plausibility.

The distinction that changes behavior: your error-detection instincts are calibrated to ordinary errors — a sign flip, an off-by-one, a unit mismatch — which leave traces. Hallucinated content is selected for looking right, because generating plausible text is what the model is optimized to do. The fabricated citation is the classic scientific case: plausible authors, plausible title, a DOI that resolves to nothing. The extraction-table case is nastier: fifty abstracts in, most rows correct, a few values from nowhere, formatted identically — nothing marks them. Consequences, verbatim from 6.1: every citation gets resolved; every quantitative claim gets verified against the data; confidence and formatting carry no evidence. A tidy table is not more likely to be correct because it is tidy. Treat LLM output like a measurement from an uncalibrated instrument: possibly excellent, unusable until you run your own calibration.

Every exercise declares its AI mode

🔒 By hand — assistant closed; the drill builds a reflex you must own

🤝 Assistant allowed — use it; verify every line; disclose the use

No badge means assistant allowed. The “by hand” syntax — indexing, slicing, core numpy and pandas — is the same syntax you later use to check the machine.

Delegate the drills and the skill never forms — and then you cannot check the assistant.

From Chapter 2 on, every exercise in the book carries one of these two badges. The logic of “by hand”: some reflexes must exist in your fingers before an assistant is useful, because they are the reflexes you verify with — a student who cannot read a slicing expression cannot review an agent’s diff. Do the by-hand drills unaided, then compare with an assistant’s answer afterward if curious. “Assistant allowed” is not “anything goes”: the verification and disclosure duties apply in full. Note the framing from the course intro: AI use here is a skill to be taught, not a shortcut to be policed — the badges are the teaching instrument.

Course law: disclose, defend, verify

• Disclose — every submission names the tools and what they did; substantial AI commits say so in the message
• Defend — any line, unaided, on request; “the AI wrote it” is not an explanation
• Verify — numbers recomputed a second way, citations resolved, interfaces checked against the docs

Undisclosed AI help is an integrity issue exactly as undisclosed human help is.

The policy, stated plainly. AI use is allowed and expected — refusing it mostly costs time — and disclosure is one or two sentences, not a form: “Claude Code drafted the loaders and first plots; I wrote the feature engineering and verified outputs against the raw files.” The defense rule has teeth: the instructor may ask anyone, at any point, to walk through their code with no assistant and no notes; if you cannot, the work is not yours yet, whatever the commit log says. Verification duties, from 1.8: never accept numerical results without recomputing or testing a known case; never accept citations without resolving them; never accept API claims without the documentation — assistants’ training data lags current libraries (this book runs numpy 2.5 and pandas 3.0). And treat AI review of your work as a first pass from a reviewer who has never done fieldwork and cannot be embarrassed — useful, not authoritative.

Today, in one table

Failure mode	Mechanism	Your counter
bad arithmetic	digits are vocabulary, not numbers	make it write and run the numpy
missed context	attention sags mid-context	name the files; spot-check what it “read”
sycophancy	tuned on human approval	ask for the case against your framing
hallucination	fabrication optimized to look right	resolve every citation; verify against data

Verification is not distrust of the tool. It is how the tool becomes usable for science.

Leave this up for questions; it is the session in one screen and it maps one-to-one onto the literature slide. Each counter becomes a graded habit: the numpy rule is today’s exercise, the citation rule is the reading arc (starting today), the framing rule is the 6.2 lab in week 4, and all four return in week 7 when students build an eval set that measures an agent’s failure rates instead of trusting spot checks. The through-line to say aloud: the same fair-evaluation discipline they will learn for models applies to the AI itself — that is the course’s distinctive move.

Now calibrate your collaborator

1. Assistant running? (Claude Code / Copilot agent — 1.9’s follow-up if not)
2. Ask it for the mean east velocity of station P395 from 1.7’s saved CSV — did it write code, or “read” the numbers? Verify the value yourself
3. Ask it for three papers on ML in your subfield — resolve every citation; count the fabrications
4. Sign up for your pulse talk (Canvas, opens today) · Reading arc stage 1 assigned — AI-assisted literature review, every citation verified, due Wed Oct 21

Ch 1 quiz: opens Tue Oct 6, closes Thu Oct 8 · Wednesday: meet the data (2.1–2.2, 1.6–1.7)

Timing for the 80-minute block (10:00–11:20), no pulse talks yet (they start week 3): deck ~25 min, exercise ~40, synthesis ~10, logistics ~5. During the exercise, circulate for task 2 — the interesting split is between students whose assistant wrote and ran code (correct behavior) and those whose assistant answered from the prompt; both outcomes teach. Synthesis: tally the task-3 fabrication counts on the board — a live measurement of hallucination rate, and quietly the class’s first eval set; keep the number for week 7. Logistics: pulse-talk sign-ups open today on Canvas (every student presents once, weeks 3–10; everyone scores every talk through the standing peer survey); reading arc stage 1 is assigned today — topic: opportunities and challenges of AI in your subfield, with the two anchor readings (Bergen et al. 2019; Karpatne et al. 2019), due Wed Oct 21 — and task 3 was a rehearsal of exactly its verification duty. Ch 1 quiz opens tomorrow on Canvas, scenario-based, closes Thursday.