6.1 From Language Models to Agents

🖥️ Lecture slides — Session 03 (Mon Oct 5)

This page gives you the working vocabulary for the rest of the chapter: what a large language model (LLM) actually computes, how one becomes an “agent,” and which failure modes come from which part of the machinery. Knowing where errors enter is what lets you check for them.

What an LLM computes¶

A large language model is a transformer — the attention-based architecture whose mechanics we covered with sequence models in Chapter 4.4 — trained to predict the next token in text. Attention lets every token condition on all the tokens before it (generation runs left to right, so the attention is causally masked: no token sees its own future), which is why these models handle long-range structure (a variable defined 200 lines earlier, a claim made three paragraphs ago) that older sequence models lost. Scaled to billions of parameters and trained on a large fraction of the public internet, next-token prediction turns out to produce a general text-in, text-out engine: summarization, translation, code, and question answering all become “continue this text plausibly.”

Three training stages turn that engine into an assistant:

1. Pretraining: predict the next token on a huge corpus. This is where knowledge of language, code, and (imperfectly, up to a training cutoff) the world comes from.
2. Instruction tuning: further training on curated (instruction, good response) pairs, so the model answers questions instead of merely continuing them.
3. Reinforcement learning from human feedback (RLHF) and its successors: train a preference model on human rankings of responses, then optimize the LLM against it. This makes models helpful and polite. It also makes them agreeable — a bias you will measure in 6.2, because a model tuned to please raters tends to please you, including when you are wrong Sharma et al., 2023.

Two mechanical facts you must keep in mind:

Context windows. The model sees a finite window of tokens — hundreds of pages in current models, but finite, and attention over very long contexts degrades: material in the middle of a long context is recalled less reliably than material near its ends Liu et al., 2024. Practical consequence for research use: an agent “reading” your 40-file repository is sampling and summarizing it, and may confidently answer from the parts it attended to.

Tokenization, and why LLMs are bad calculators. Text is split into tokens — subword chunks, not characters and not words. Numbers get fragmented arbitrarily: 13.847 may become 13, ., 847. The model has no numeric type; digits are just vocabulary. So arithmetic is pattern completion over digit strings, which works for common patterns and fails silently elsewhere. The same applies with more force to numeric series: pasting 3,650 daily GNSS displacements into a prompt and asking for the trend hands a regression problem to a text-completion engine. It will answer, fluently, and the number will be roughly plausible. The fix is not a better prompt; it is giving the model a calculator — which is the entire point of tool use, below. A well-built agent asked for a GNSS velocity should write and run numpy code, not “read” the numbers.

Retrieval-augmented generation (RAG)¶

Pretraining knowledge is frozen at a cutoff and blurry at the level of specifics. RAG bolts on a memory: documents are split into chunks and embedded (Chapter 3’s representation ideas), the user’s question retrieves the most similar chunks, and the retrieved text is placed in the context window with the instruction “answer from these documents.” The model now cites material it was never trained on.

Geoscience example: a RAG system over your network’s station metadata corpus (instrument responses, deployment logs, data problem reports). “Which broadband stations near the Cascadia margin had known timing issues in 2021?” becomes retrievable and answerable with sources. The failure mode moves too: a RAG answer is only as good as its retrieval, and the model will happily synthesize an answer from the wrong chunks. When you evaluate a RAG system, evaluate the retrieved sources, not just the prose.

Tool use: what makes an agent¶

Function calling lets a model emit, instead of prose, a structured request — query_catalog(min_magnitude=5, region="Cascadia", since="2020-01-01") — that your code executes, returning the result to the model’s context. An agent is nothing more exotic than:

an LLM + a set of tools + a loop over observations.

The model proposes an action, the harness executes it, the observation goes back into the context, and the model proposes the next action, until it decides the task is done. Reading files, running Python, searching the web, and querying an FDSN earthquake catalog are all just tools in the loop. An agent asked “did seismicity rate change after the 2022 M6.4 event?” can query the catalog, write the rate-change test, run it, look at the numbers, and revise — a genuine multi-step scientific task.

This architecture relocates trust. The arithmetic is now done by numpy, which is reliable. What remains unreliable is everything the model still decides: which data to query, which test to run, whether the completeness magnitude was handled, whether an empty query result means “no earthquakes” or “wrong request.” Agent errors are usually not calculation errors; they are specification and judgment errors. That is why 6.3 evaluates agents end to end against ground truth instead of reviewing their code line by line.

One paragraph on reasoning models and test-time compute: current models can be run so that they generate long internal chains of intermediate work — drafting, checking, backtracking — before answering, spending more compute per query in exchange for better performance on math, code, and multi-step problems. This reliably helps, and it changes the cost model (a hard question costs more than an easy one). It does not change your obligations: a wrong answer after ten thousand tokens of deliberation is wrong, and long visible “reasoning” can make errors more persuasive, not less common.

Open-weight vs. API models¶

You can call a hosted frontier model through an API, or run an open-weight model (weights downloadable, run on your own hardware) locally. The trade-offs are generic and worth reasoning through per project rather than by fashion:

For batch scientific pipelines — structured extraction from ten thousand abstracts, say — a small open-weight model with a well-designed eval set often beats an expensive frontier model with none.

Hallucination: a failure mode distinct from error¶

An ordinary error is a wrong answer produced by a traceable process — a bug, a bad assumption, noise. Hallucination is fluent fabrication: the model produces specific, confident, well-formatted content with no source at all, because generating plausible text is exactly what it is optimized to do. The distinction matters because your error-detection instincts are calibrated to ordinary errors, which usually look wrong somehow. Hallucinated content is selected for looking right.

The canonical scientific case is the fabricated citation: real-sounding authors, a plausible title, a real journal, a well-formed DOI that resolves to nothing. Structured extraction has a subtler version — ask a model to pull magnitudes, depths, and locations from fifty abstracts into a table, and most rows will be right while a few contain values that appear nowhere in the source text, formatted identically to the correct ones. Nothing marks them.

The consequences, which the next two notebooks turn into practice:

• Every citation gets checked — resolve the DOI, confirm the paper exists, confirm it says what it is cited for. No exceptions; the cost of checking is minutes, and 6.2 makes you design the procedure.
• Every quantitative claim gets verified against the data, not against plausibility. Plausibility is the one test hallucinations are guaranteed to pass.
• Confidence and formatting carry no evidence. A table is not more likely correct because it is a tidy table.

Treat LLM output the way you treat a measurement from an uncalibrated instrument: possibly excellent, unusable until you have run your own calibration. Building that calibration is the next two notebooks.