But Does It Work? — Nine Commits

Post 3 finished with an agent that answered its question forty-one times out of forty-two, and a grader that could not tell you why. The check was a case-insensitive substring: put agent/src/config.ts and gpt-5-mini in the final sentence and the run is a success. Twenty-one runs passed it. All twenty-one also had the ground truth and a published post about the ground truth in the same set of search results, and a sentence naming the file and the value is consistent with either.

So the last post said, in public, that it was handing the grading problem here. This is that.

What I built

The trace already knows. Every recorded run in this repository is a list of frames, and the union is short enough to read in one go — a question, a tool call with its arguments, the result that came back, a step of text, and the two frame types the later commits will need. Nothing in it is prose about what happened. It is what happened, including the parts the model did not mention afterwards.

An eval that reads a trace is therefore already an eval that reads a decision. What it needed was one distinction the schema does not draw, because the schema does not care and the grader has to:

export type Observation =
  | { frame: number; tool: string; kind: 'path'; file: string }
  | { frame: number; tool: string; kind: 'line';
      file: string; line: number; text: string }

list_files returns names. search_files returns lines. A name tells the model that a file exists; a line tells it what is written inside one. By the time the answer is written both are just tokens in the context, which is exactly why a check over the answer cannot separate them — and separating them turns out to be most of the value.

On top of that, three grades per run instead of one.

The answer grade is the recorder’s own rule, recomputed from the committed file rather than from the live run: all the expected facts present is a success, some is partial, none is a failure. It is here to be compared against, not to be believed. A test asserts that it agrees with the outcome each trace was written with, on all six runs there are facts to check — post 1’s question has no fact a substring could check, so it grades ungraded and the test skips it — so that when the two columns diverge it is the run that moved and not the reimplementation.

Unread claims is the first thing the frames can say that the sentence cannot. For each fact the answer states, did any tool actually return it? A fact that appears in a line is read. A fact that appears only as a file name in a listing is named-only — the model inferred it. A fact that appears in nothing at all is unsupported; it came out of the model. Anything in the answer that is not read is an unread claim, and the substring grader gives it full marks.

Provenance is the axis post 3 asked for, and the useful part of it is the verdict that admits defeat:

export type Provenance =
  | 'grounded'     // the ground-truth line was observed, no copy of it was
  | 'undecidable'  // both were observed; the trace cannot say which was used
  | 'from-copy'    // a copy was observed, the ground truth was not
  | 'unevidenced'  // neither was observed
  | 'ungradable'   // the question has no single line for an answer

A copy is any line outside the source file carrying the value the answer turns on. That rule over-counts on purpose: it flags a line that restates the value without naming the file, which could not by itself have produced the answer. A grader whose mistakes all run toward I cannot tell is one you can act on. One whose mistakes run toward grounded is the thing this post exists to complain about.

Why this is vitest and not an eval framework

The ai SDK’s ecosystem has an eval story and the course this series follows uses Laminar, and I used neither, because the thing being graded here is a pure function from a committed JSON file to a verdict. No model is called. No score is averaged. Nothing is non-deterministic, nothing needs a dashboard, and nothing needs to be uploaded anywhere to be looked at later. An eval framework earns its keep when runs are live and scores are noisy — which is post 5, and post 5 can have one. Here the unit under test is a function, the repository already runs vitest in CI, and adding a service dependency to a check whose entire virtue is that it runs offline from a clean checkout would be paying for the opposite of what I want.

So the suite is tools/eval/, the gate rides in npm test, and there is a report:

npm run eval           # one block per run
npm run eval -- --tsv  # the same thing as a table

Each committed trace has a row in tools/eval/suite.ts saying which question it was answering and what this eval says about it. Those expectations were written down as predictions, from reading the traces, before the grader was run over them; all seven matched on the first run and none has been changed since. That ordering matters, because the other ordering — run it, then write down whatever it said — produces a suite that agrees with everything.

What broke

What post 2’s `partial` was actually worth

Post 2’s precise run is the one that called list_files, got the whole project back, and then said it could not read the project. The recorder graded it partial, and post 2 defines a partial as a run whose answer carried one of the two facts and not the other. The fact it carried was the file.

This eval counts 68 observations in that trace and every one of them is a file name. Not one line of file contents was ever returned to the model. So the fact it was credited with is an unread claim, and here is the sentence it was credited for:

The file is likely agent/src/config.ts or agent/src/run.ts — try opening agent/src/config.ts (cat agent/src/config.ts) to see the model name declaration.

That is two guesses and an instruction to go and check. The substring grader sees the first guess, finds one of its two strings, and awards half. Post 2 called its own check “the crudest possible” and handed grading here, so this is a prediction coming true rather than a claim being overturned — no number in post 2’s tables changes. What changes is what partial is worth in the column it was counted in.

Not one run of this series’ question can be shown to have answered from the source

Five recorded runs across posts 2, 3 and this one ask the same question, character for character. The grader can return grounded. It returns it for none of them:

run	answer	what the frames say
2 · `trace-a-precise`	partial	unevidenced, 1 unread claim
2 · `trace-b-thin`	failure	unevidenced
3 · `trace`	success	undecidable, 1 copy
3 · `trace2-corpus-hit`	success	undecidable, 16 copies
4 · `trace-a-model-name`	success	undecidable, 1 copy

Three of those runs end with this sentence:

The model name is set in agent/src/config.ts — it exports MODEL_NAME = 'gpt-5-mini'.

Not approximately. The same string, three times, from three separate runs across two commits:

for t in site/src/content/posts/03-the-loop/trace.json \
         site/src/content/posts/03-the-loop/trace2-corpus-hit.json \
         site/src/content/posts/04-does-it-work/trace-a-model-name.json; do
  node -e "console.log(require('./$t').frames.filter(f=>f.type==='assistant').at(-1).content)"
done | sort -u | wc -l

That prints 1. What it prints is 86 bytes of UTF-8 — eighty-four characters, one of them an em dash. Behind those three identical sentences are three different result sets, with one, sixteen and one copies of the answer in them. A grader reading the sentence has nothing to work with. It is looking at the same bytes each time, and it is right to give the same grade, and the grade is worthless.

A contrast post 3 drew is weaker than it reads

Post 3 puts two runs on the page: the demonstration of the loop at the top, and tally-06 further down as the run that was handed the answer key. Of the first it says it never saw any of the prose — and I want to be exact about what that sentence claims, because I drafted a correction to it that it does not deserve.

It sits under the heading The corpus now contains posts about the answer, four lines after post 3 works out that the run’s search hit the forty-match cap fourteen files in and — post 3’s words — never got as far as site/. That is luck, not hygiene. So “the prose” there means the published posts, and on its own terms the claim is true: 0 of that run’s 40 matches are under site/. Post 3 also reaches this post’s conclusion before this post does, in the same section: I cannot tell you it did not happen, because config.ts was in the same result set every single time … The recorder’s substring check cannot tell “read the source” from “read the post about the source”, and neither can I.

The eval does not disagree with any of that. It agrees and puts a number on it. What it adds is one thing the framing of those two runs leaves standing. The featured run’s results are not free of the answer: they contain agent/src/cli.ts line 48, which post 3 quotes from that very trace two sections earlier — step to frame five and read the third match — and describes as naming both graded facts in one string. One line is enough. So the distance between the two runs is sixteen copies against one, which is a difference of degree, and both verdicts come out undecidable. The contrast is real as contamination. It is not a contrast of provenance, which is what a reader flipping between the two players might take it for.

Two counts of the same trace, and they do not conflict. Post 3 counts the TRACE_EXPECT-shaped pair — path and value on one line with a comma between them — and gets five lines in four files, which is exact; I checked it. This eval counts any line outside config.ts carrying the value at all, and gets sixteen in eight, because it deliberately includes lines that hand over the value without naming the file. Two rules, one trace, both printed.

Nothing in post 3 needs editing. This is a reading of it, and it goes here because that is where this series puts its readings.

A fixture caught a bug the whole corpus would have hidden

The first version of the support check asked whether a fact appeared in the text of a line. But a search match arrives as a file, a line number and the line, all at once, so a run that reads agent/src/config.ts:1 has read the path as surely as it has read the assignment. The first version called that file fact unsupported.

All seven committed traces graded identically before and after the fix, because in this repository the path is also spelled out inside other lines, so the wrong rule reached the right answer for the wrong reason on every real run I have. The thing that caught it was a fixture about a fictional project with a port number in it, written deliberately against a question this project does not ask. An eval tested only against the corpus it was built for cannot tell you whether it is measuring that corpus or remembering it.

The control retires the moment this page is published

The second run below asks about something this repository stated in exactly one place at v3-the-loop, which is the tree both runs below were recorded against and the only reason one of them can be graded grounded. This paragraph ends that. The extension list is now in a published post, the post is in the corpus, and the next run to ask that question gets a copy back alongside the source and an undecidable with it.

That is the third time this series has spent a measurement by describing it, and it is beginning to look less like an accident than like a property. An experiment run against the repository that documents it has a half-life.

The demo

Same question as posts 2 and 3. Both runs below carry v4-does-it-work as their commit, which is the tag this post is bound to, but the tree they actually walked was v3-the-loop — they were recorded before a line of tools/eval/ existed, so nothing the grader says about them can have been arranged by writing the grader.

Recorded run

Which file in this project sets the model name the agent uses, and what is it set to?

Recorded from gpt-5-mini at commit v4-does-it-work. The run finished the task.

Frame 1 of 4

204 tokens in context, against this run's peak of 1,908.

You
Which file in this project sets the model name the agent uses, and what is it set to?

The same question, one commit later

Which file in this project sets the model name the agent uses, and what is it set to?

Recorded from gpt-5-mini at commit v4-does-it-work. The step-through needs JavaScript, so the whole run is printed here instead.

You
Which file in this project sets the model name the agent uses, and what is it set to?

Calls search_files

{
  "pattern": "model|MODEL|model_name|MODEL_NAME|gpt|gpt-4|gpt-4o|gpt-4o-mini|gpt-4o-mini-1|gpt-4o-realtime-preview|gpt-4o-mini-tts|gpt-4o-mini-transcribe|gpt-3.5|gpt-3.5-turbo|text-davinci|openai|ChatCompletion|chat.completions|ChatCompletion.create|ChatCompletion.create|ChatCompletion.create\\(|openai.ChatCompletion|openai.ChatCompletion.create|agent|Agent|AGENT"
}

Returns

{
  "ok": true,
  "pattern": "model|MODEL|model_name|MODEL_NAME|gpt|gpt-4|gpt-4o|gpt-4o-mini|gpt-4o-mini-1|gpt-4o-realtime-preview|gpt-4o-mini-tts|gpt-4o-mini-transcribe|gpt-3.5|gpt-3.5-turbo|text-davinci|openai|ChatCompletion|chat.completions|ChatCompletion.create|ChatCompletion.create|ChatCompletion.create\\(|openai.ChatCompletion|openai.ChatCompletion.create|agent|Agent|AGENT",
  "filesSearched": 10,
  "matches": [
    {
      "file": "agent/package.json",
      "line": 2,
      "text": "\"name\": \"@nine-commits/agent\","
    },
    {
      "file": "agent/package.json",
      "line": 9,
      "text": "\"@ai-sdk/openai\": \"^4.0.69\","
    },
    {
      "file": "agent/src/cli.ts",
      "line": 13,
      "text": "* its own log from here on, and it lives in `agent/traces/` — gitignored, and"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 14,
      "text": "* outside what the agent's own tools can see, so a tally can never become part"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 20,
      "text": "* v3 adds `steps` and `stopped_by`. Once the agent can go round again, how"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 35,
      "text": "* Read and checked BEFORE the model is called. The recorder refuses to grade a"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 48,
      "text": "'must contain, e.g. TRACE_EXPECT=\"agent/src/config.ts,gpt-5-mini\". ' +"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 63,
      "text": "console.error('usage: npm start --workspace @nine-commits/agent -- \"<task>\"')"
    },
    {
      "file": "agent/src/cli.ts",
      "line": 75,
      "text": "model: result.model,"
    },
    {
      "file": "agent/src/config.ts",
      "line": 1,
      "text": "export const MODEL_NAME = 'gpt-5-mini'"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 19,
      "text": "* the answer they were graded against. The agent's tools read this repository,"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 53,
      "text": "model: 'a-model',"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 8,
      "text": "model: string"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 22,
      "text": "/** The last thing the model actually said, across however many steps it took. */"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 35,
      "text": "* — a single model call cannot act, so no v1 run could finish a task — and it"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 36,
      "text": "* stops being true the moment the agent has hands. A badge under a trace that"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 82,
      "text": "*   own input count. That is why the same run costs more before the model has"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 126,
      "text": "model: input.model,"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 33,
      "text": "it('reads the model when the last step asked for no tool', () => {"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 34,
      "text": "expect(whatStopped([step('tool-calls'), step('stop')])).toBe('model')"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 37,
      "text": "it.each(NOT_TOOL_CALLS)('treats %s as the model stopping', (reason) => {"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 38,
      "text": "expect(whatStopped([step(reason)])).toBe('model')"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 48,
      "text": "* the model. It is not a state `generateText` produces; the branch exists so"
    },
    {
      "file": "agent/src/run.test.ts",
      "line": 52,
      "text": "expect(whatStopped([])).toBe('model')"
    },
    {
      "file": "agent/src/run.ts",
      "line": 2,
      "text": "import { openai } from '@ai-sdk/openai'"
    },
    {
      "file": "agent/src/run.ts",
      "line": 3,
      "text": "import { INSTRUCTIONS, MODEL_NAME } from './config.ts'"
    },
    {
      "file": "agent/src/run.ts",
      "line": 14,
      "text": "/** Context the model was handed for this step. */"
    },
    {
      "file": "agent/src/run.ts",
      "line": 16,
      "text": "/** That context plus what the model produced from it. */"
    },
    {
      "file": "agent/src/run.ts",
      "line": 23,
      "text": "* `model` is the one that is meant to fire: a step came back asking for no"
    },
    {
      "file": "agent/src/run.ts",
      "line": 25,
      "text": "* `step-cap` is the backstop firing instead, and it means the model was still"
    },
    {
      "file": "agent/src/run.ts",
      "line": 28,
      "text": "export type StopReason = 'model' | 'step-cap'"
    },
    {
      "file": "agent/src/run.ts",
      "line": 31,
      "text": "model: string"
    },
    {
      "file": "agent/src/run.ts",
      "line": 40,
      "text": "* What ends a run is the model declining to call a tool: the SDK goes round"
    },
    {
      "file": "agent/src/run.ts",
      "line": 55,
      "text": "* model. With these two tools that something is the cap, and only because the"
    },
    {
      "file": "agent/src/run.ts",
      "line": 67,
      "text": "return steps.at(-1)?.finishReason === 'tool-calls' ? 'step-cap' : 'model'"
    },
    {
      "file": "agent/src/run.ts",
      "line": 81,
      "text": "* v3: call the model, run whatever tool it asked for, append the call and the"
    },
    {
      "file": "agent/src/run.ts",
      "line": 87,
      "text": "* the second, so there was never a turn in which the model could act on what"
    },
    {
      "file": "agent/src/run.ts",
      "line": 94,
      "text": "model: openai(MODEL_NAME),"
    },
    {
      "file": "agent/src/run.ts",
      "line": 118,
      "text": "model: MODEL_NAME,"
    },
    {
      "file": "agent/src/tools/descriptions.ts",
      "line": 6,
      "text": "* versions of the code. Everything else — model, instructions, task, the tools"
    }
  ],
  "truncated": true
}

Model
The model name is set in agent/src/config.ts — it exports MODEL_NAME = 'gpt-5-mini'.

One search, forty matches, the answer in the first sweep, and the sentence again. agent/src/config.ts:1 is in there. So is agent/src/cli.ts:48. Verdict: undecidable. The loop works, the answer is correct, and the run cannot prove it.

Then the control — a question this repository answers in exactly one line:

Recorded run

Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

Recorded from gpt-5-mini at commit v4-does-it-work. The run finished the task.

Frame 1 of 12

209 tokens in context, against this run's peak of 3,094.

You
Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

A question with one answer in the corpus

Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

Recorded from gpt-5-mini at commit v4-does-it-work. The step-through needs JavaScript, so the whole run is printed here instead.

You
Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

Calls search_files

{
  "pattern": "binary|isBinary|is_binary|BINARY|isBinaryFile|is_text_file|binaryExtensions|binary_extensions|isBinaryFileSync|EXCLUDE|excluded|should not be searched|isBinaryFile|isBinaryPath|file is binary|skip binary|binary file|isBinaryFile|isBinary|isBinaryPath|isBinaryFileSync|binaryExtensions|binaryExt|isText|text|searchable"
}

Returns

{
  "ok": true,
  "pattern": "binary|isBinary|is_binary|BINARY|isBinaryFile|is_text_file|binaryExtensions|binary_extensions|isBinaryFileSync|EXCLUDE|excluded|should not be searched|isBinaryFile|isBinaryPath|file is binary|skip binary|binary file|isBinaryFile|isBinary|isBinaryPath|isBinaryFileSync|binaryExtensions|binaryExt|isText|text|searchable",
  "filesSearched": 15,
  "matches": [
    {
      "file": "agent/src/cli.ts",
      "line": 70,
      "text": "console.log(result.steps.at(-1)?.text ?? '')"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 8,
      "text": "text: '',"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 42,
      "text": "text: `It is set in ${FILE}, to ${VALUE}.`,"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 85,
      "text": "it('reports context size per frame, not deltas', () => {"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 87,
      "text": "// The question already sits in a context holding the system instruction"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 97,
      "text": "steps: [step({ text: VALUE, inputTokens: 900, totalTokens: 950 })],"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 113,
      "text": "text: `${VALUE} is set in ${FILE}`,"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 151,
      "text": "step({ text: `${FILE} sets it to ${VALUE}.` }),"
    },
    {
      "file": "agent/src/recorder.test.ts",
      "line": 161,
      "text": "const shouted: RunStep[] = [step({ text: 'IT IS PORT 4321.' })]"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 25,
      "text": "const text = steps[i]?.text ?? ''"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 26,
      "text": "if (text.trim() !== '') return text"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 76,
      "text": "* `tokens` is one entry per frame, and it is the size of the context at that"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 80,
      "text": "* - the user's question sits in a context that already holds the system"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 117,
      "text": "if (step.text.trim() !== '') {"
    },
    {
      "file": "agent/src/recorder.ts",
      "line": 118,
      "text": "frames.push({ type: 'assistant', content: step.text })"
    },
    {
      "file": "agent/src/run.ts",
      "line": 11,
      "text": "text: string"
    },
    {
      "file": "agent/src/run.ts",
      "line": 14,
      "text": "/** Context the model was handed for this step. */"
    },
    {
      "file": "agent/src/run.ts",
      "line": 16,
      "text": "/** That context plus what the model produced from it. */"
    },
    {
      "file": "agent/src/run.ts",
      "line": 102,
      "text": "text: step.text,"
    },
    {
      "file": "agent/src/tools/descriptions.ts",
      "line": 29,
      "text": "'Search the text inside the project files for a regular expression. ' +"
    },
    {
      "file": "agent/src/tools/fs.test.ts",
      "line": 67,
      "text": "expect(hit?.text).toContain('MAX_STEPS')"
    },
    {
      "file": "agent/src/tools/fs.test.ts",
      "line": 73,
      "text": "// the assignment line in `agent/.env` if that file were ever searchable."
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 41,
      "text": "/** Files whose bytes are not text, so searching them is noise. */"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 42,
      "text": "const BINARY = /\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 85,
      "text": "export type SearchMatch = { file: string; line: number; text: string }"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 125,
      "text": "/** Every line in every text file that matches `pattern`, with its file and line number. */"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 149,
      "text": "if (BINARY.test(file)) continue"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 169,
      "text": "text: line.trim().slice(0, LINE_LIMIT),"
    },
    {
      "file": "DECISIONS.md",
      "line": 75,
      "text": "credential-rule over-match on prose is ACCEPTED, not narrowed. On a security boundary, over-redacting prose is the correct direction to err, and once G1 is fixed it degrades content rather than failin"
    },
    {
      "file": "DECISIONS.md",
      "line": 79,
      "text": "ESCALATE EARLY to a fresh implementer on a more capable model for round 3, rather than resuming the original for its third round as the default allows. Two consecutive rounds patched the measured symp"
    },
    {
      "file": "DECISIONS.md",
      "line": 83,
      "text": "round 3 requires a GENERATED test corpus (secrets x wrappers x contexts, several hundred cases) instead of an enumerated table. Both prior rounds passed green because the tests only ever re-checked th"
    },
    {
      "file": "DECISIONS.md",
      "line": 91,
      "text": "round 4 RESUMES the round-3 implementer instead of dispatching a fresh one as the rounds-4-5 rule prescribes. Two reasons: no model tier above opus is available here, so the capability-bump half of th"
    },
    {
      "file": "DECISIONS.md",
      "line": 119,
      "text": "the reviewer found F1 SURVIVING in validate-traces.ts. The plan wrote that file before F1 was discovered in normalize.ts, so it still scans findSecrets(JSON.stringify(trace)) - the exact pattern norma"
    },
    {
      "file": "DECISIONS.md",
      "line": 195,
      "text": "the design refuses the conventional AI aesthetic on purpose. The user asked for something that \"remete AI\", and the obvious reading is gradients, glow and neural-net imagery. Decided against it: the s"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 17,
      "text": "- **Published site language is English.** Every string that reaches a reader — post prose, UI labels, alt text, LinkedIn copy, README — is English. Code comments are English too."
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 409,
      "text": "it('leaves innocent text untouched', () => {"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 435,
      "text": "it('returns nothing for clean text', () => {"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 815,
      "text": "`RunResult = { text: string; model: string; totalTokens: number }`"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 958,
      "text": "text: string"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 975,
      "text": "text: result.text,"
    }
  ],
  "truncated": true
}

Calls search_files

{
  "pattern": "const BINARY = /\\\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i"
}

Returns

{
  "ok": true,
  "pattern": "const BINARY = /\\\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i",
  "filesSearched": 82,
  "matches": [],
  "truncated": false
}

Calls search_files
```
{
  "pattern": "BINARY.test(file)"
}
```

Returns

{
  "ok": true,
  "pattern": "BINARY.test(file)",
  "filesSearched": 82,
  "matches": [],
  "truncated": false
}

Calls search_files

{
  "pattern": "Files whose bytes are not text, so searching them is noise."
}

Returns

{
  "ok": true,
  "pattern": "Files whose bytes are not text, so searching them is noise.",
  "filesSearched": 82,
  "matches": [
    {
      "file": "agent/src/tools/fs.ts",
      "line": 41,
      "text": "/** Files whose bytes are not text, so searching them is noise. */"
    }
  ],
  "truncated": false
}

Calls search_files

{
  "pattern": "agent/src/tools/fs.ts"
}

Returns

{
  "ok": true,
  "pattern": "agent/src/tools/fs.ts",
  "filesSearched": 82,
  "matches": [
    {
      "file": "site/src/content/posts/02-hands/trace-a-precise.json",
      "line": 45,
      "text": "\"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/03-the-loop/trace.json",
      "line": 48,
      "text": "\"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/03-the-loop/trace.json",
      "line": 295,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    }
  ],
  "truncated": false
}

Model
The check is in agent/src/tools/fs.ts. It uses this regex:
/\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i
So it treats these extensions as binary: .png, .jpg, .jpeg, .gif, .ico, .webp, .avif, .woff, .woff2, .ttf, .otf, .eot, .pdf, .zip.

Five searches for one question. The first sweep already contained agent/src/tools/fs.ts line 42, which is the whole answer; the model then searched for the matched line verbatim and got nothing back twice, because it was searching for a regular expression using its own text as the pattern. Then it searched for the doc comment above it, got one hit, searched for the file path, and answered. Four of its five calls were the model checking a thing it had already been told, and two of those four came back with nothing.

None of that is visible in the final sentence. All of it is in the trace. Verdict: grounded — the source line observed, no copy of it anywhere in what came back.

Which is the finding under the finding. The model-name question is undecidable in this corpus and no grader fixes that, because the trace does not record which of the forty matches the model attended to and the API does not expose it. What changed the verdict was not a better grader. It was a question the corpus answers once.

All seven traces in the repository, graded twice, are in verdicts.tsv, generated by npm run eval -- --tsv, with each run’s final answer in the last column so the identical-sentence claim above is something you can check rather than something you have to take from me.

It goes red

An eval suite that cannot fail is the defect this project has already shipped once: post 1’s leak gate had never seen a real trace and would have reported success over a leak, because a gate that finds nothing and a gate that is not looking print the same line.

So, three ways this one fails.

Claim that post 3’s demonstration run answered from the source — change its one expectation in tools/eval/suite.ts from undecidable to grounded — and the gate says so, and exits 1:

03-the-loop/trace.json
  tools        list_files > search_files
  observations 117
  answer       success (recorded: success)
  unread       (none)
  provenance   undecidable (1 ground truth, 1 copies)
  MISMATCH     provenance: expected grounded, got undecidable

1 of 7 case(s) no longer match their recorded verdict:
  03-the-loop/trace.json: provenance: expected grounded, got undecidable

Second, tools/eval/corpus.test.ts takes the real committed corpus-hit trace and removes matches from its search result in memory — nothing is written and no altered trace is shown to anyone as a run — and asserts that the verdict moves to grounded with the copies gone, to from-copy with the source gone, and to unevidenced with both gone, while the answer grade stays success through all four, because the sentence never changed. That is post 3’s complaint, reproduced on demand.

Third, and this is the one I would actually rely on: a tool result whose shape the eval does not recognise raises rather than returning nothing. An unrecognised result would contribute zero observations, every claim would grade unsupported, and the suite would look most rigorous at the exact moment it had stopped reading. Post 6 changes the tool set; this is the line that will notice.

What this does not fix

The thesis at the top of this page says a single-turn eval tests the decision rather than the prose, and it survives, with a condition it did not come with: a decision eval can only distinguish what the run recorded. Where the ground truth and a copy of it arrive in the same tool result, the decision the model made is genuinely not in the trace, and a grader that reported grounded there would be inventing. The three ways out are a question the corpus answers once, which is the run above and which publishing retires; an API that reports which part of a result the model attended to, which does not exist; and deleting the copies, which this series has forbidden itself twice in writing.

So I have left series.ts alone. Post 3 changed its thesis because the diff contradicted it. Nothing here contradicts this one — it just arrives with a limit printed on the side.

The commit

There are now two graders in this repository. One reads the sentence, and on four of the seven traces in the tree it has been cheerfully, publicly overgenerous. The other reads the frames and says that three of those four are undecidable and the fourth was credited for a guess. On the other three — post 1’s chatbot, post 2’s thin run, and the control above — the substring check is not overgenerous about anything, which is worth saying too. A crude grader is not a grader that is always wrong. It is a grader that cannot tell you when it is.

Neither of them is judging quality. Both are checking a run against facts I wrote down before it started, which works precisely as long as the answer is a string I already know. The next question does not have one of those, and then the grader has to be a model too.

The next commit is about judging a moving target.

What I built

Why this is vitest and not an eval framework

What broke

What post 2’s partial was actually worth

Not one run of this series’ question can be shown to have answered from the source

A contrast post 3 drew is weaker than it reads

A fixture caught a bug the whole corpus would have hidden

The control retires the moment this page is published

The demo

It goes red

What this does not fix

The commit

What post 2’s `partial` was actually worth