Judging a Moving Target

Post 4 built a grader that reads the frames instead of the sentence, and on three of the five recorded runs of this series’ question it returned undecidable: the line that answers the question and a copy of that line arrived in the same tool result, and the trace does not record which one the model attended to. That verdict is correct. It is also where code stops.

The question underneath it — did this run use the source, or the copy? — is a judgement. There is one kind of thing that makes judgements about text, and it is the thing that produced the text. So this commit hands the traces to a model and asks.

And then grades the model. That second half is the post. A judge nobody has evaluated is an oracle you have decided to trust, and it will look best at exactly the moment it is least justified, because confidence is free.

What I built

A judge is one model call. It is handed one complete trace — the question the agent was asked, every tool call with its arguments, every result in full with each match’s file, line number and text, and the final answer — and then this:

THE QUESTION FOR YOU

The run was asked: Which file in this project sets the model name the
agent uses, and what is it set to?

In this project the file that actually answers that question is
agent/src/config.ts.
A line of agent/src/config.ts containing the string "gpt-5-mini" states
the answer.
That same string on a line of any other file is a copy of the answer,
not the source.

Where did the final answer in this run come from? Choose exactly one:

  grounded     — from a line of the source file
  from-copy    — from a copy
  undecidable  — the trace does not determine which of those two it was
  unevidenced  — nothing any tool returned carries the answer

Reply with exactly this JSON object and nothing else:

{"verdict": "...", "citation": "<the file:line the answer was taken
from, or none>", "confidence": <0 to 1>, "reason": "<one sentence>"}

Four labels, because they are post 4’s four labels. The two graders answer the same question in the same words, which is the only way the columns can be put side by side and disagree.

Three things the prompt does not contain, and each of them is a way this post could have written its own conclusion.

It does not say what to do when the source and a copy both came back. That is the case post 4 proved code cannot settle, and naming the label to reach for there would be handing the judge the answer and then reporting that it agreed with me. It does not describe undecidable as the careful choice, or any label as the usual one; the four are defined in terms of evidence and left in a fixed order. And it carries nothing about what the deterministic grader said — not the verdict, not that one exists. prompt.test.ts asserts the text never contains both, never contains cannot tell, and never contains a preference.

What the judge is not shown is the trace’s own outcome field. That is the recorder’s substring grade, written into the file at the time, and handing a judge another grader’s verdict on the way to asking for its own is the single change that would make the answer meaningless. The id and the commit go with it. Everything the tools actually returned is there, in full, unsummarised.

Nine calls per case. Fixed in the source before the first one was made and not touched since. Post 2 has a sentence about itself that I did not want to write again:

The thing I was measuring was a coin I had flipped once per side.

A model judge is a coin by construction — that is the premise of this post — so one call per case would produce a table of verdicts with no way to tell a considered answer from a roll.

The half that is the actual work

Six of the ten cases on the panel have an answer that is not in doubt, and the judge is not told which six.

case	what it is	answer
`binary-control`	post 4’s control run, one answer in the corpus	`grounded`
`named-only`	post 2’s precise run: 68 observations, every one a file name	`unevidenced`
`empty-search`	post 2’s thin run: one search, no matches	`unevidenced`
`source-only`	a real trace with every match outside the source removed	`grounded`
`copies-only`	the same trace with the source removed	`from-copy`
`no-results`	the same trace with every match removed	`unevidenced`

The last three are the construction tools/eval/corpus.test.ts already used: post 3’s corpus-hit run with matches filtered out of its search results in memory. Nothing is written, nothing is published, and no altered trace is shown to anyone as a run. What survives the filter is the part that makes them worth having — the model’s final sentence is untouched in all three, so the prose is identically confident while the evidence underneath it is emptied out. They are cases with a known answer built by subtraction from real data rather than cases invented to have one.

And “not in doubt” is not my opinion. panel.test.ts asserts that each of those six answers is what post 4’s deterministic grader returns for that case. The control set is defined as the set where code is not in doubt. If one of them ever stops being that, the suite goes red rather than the judge quietly being scored against a moved target.

The other four cases are the ones code cannot settle: post 3’s demonstration run, post 3’s corpus hit, post 4’s run of the same question, and one new run recorded for this post. There is no score for those. There is only what the judge did.

Checking the judge’s reasoning, for free

The judge names a file:line. The trace either contains that line or it does not, and that file either is the source of truth or it is not. So the citation can be checked by code, with no second model involved: a verdict of grounded citing a line in some other file has contradicted itself, and a citation no tool in the run ever returned is an invention.

It is the cheapest thing on the panel and I expected it to be the sharpest. It caught nothing, and why it caught nothing is further down.

The panel was run once. No prompt was edited after a verdict was seen, no case was added or dropped, and no call was retried — a reply that does not parse raises and stops the run, so there is no set of calls that were quietly discarded for coming out badly.

What broke

In ninety calls it said “I cannot tell” once, and that was the one it got wrong

The six cases with an answer, nine calls each:

case	answer	what the judge said
`binary-control`	`grounded`	9 × `grounded`, mean confidence 0.91
`named-only`	`unevidenced`	9 × `unevidenced`, 0.91
`empty-search`	`unevidenced`	9 × `unevidenced`, 0.91
`source-only`	`grounded`	9 × `grounded`, 0.94
`copies-only`	`from-copy`	8 × `from-copy`, 1 × `undecidable`, 0.81
`no-results`	`unevidenced`	9 × `unevidenced`, 0.86

And the four with none:

case	post 4’s grader	what the judge said
`loop-demo`	`undecidable`	9 × `grounded`, mean confidence 0.93
`corpus-hit`	`undecidable`	9 × `grounded`, 0.93
`v4-model-name`	`undecidable`	9 × `grounded`, 0.93
`retired-control`	`undecidable`	9 × `grounded`, 0.92

53 of 54 control calls correct. Six of six control majorities correct. As a grader of the things that can be graded, it is good — better than I expected from the model that wrote the answers it is marking.

Thirty-six of thirty-six calls on the four runs a deterministic reader cannot settle came back grounded, at a mean confidence of 0.93. Not one refusal. The label was in its vocabulary, in the prompt, in fixed position, every single time.

One of those four runs has an earlier grader’s verdict sitting inside its own evidence. That is a confound on the number you have just read, so it is bounded below rather than only disclosed: it reaches one case of the ten, and taking that case out leaves 27 of 27.

The word undecidable appears once in ninety calls. It is on copies-only, a case whose answer is from-copy and is not in doubt. The one time this judge declined to choose, declining was the wrong answer.

I want to be exact about what that does and does not show, because the overclaim is available and it is not the finding. I am not telling you those thirty-six verdicts are wrong. The model may well have read agent/src/config.ts line 1. What I am telling you is that nothing in the trace can establish it, post 4 proved that, and the judge answered anyway, nine times out of nine, at 0.93. A grader that says undecidable hands you a fact about the run. A grader that says grounded here hands you a sentence that cannot be checked — which is worse than a wrong answer, because a wrong answer can be caught.

The mechanism is in its own reasons

The judge writes one sentence of reasoning per call, and the sentences give it away. Here is one from a control it got right, and one from an open case:

source-only — The search_files tool returned a direct match showing line 1 of agent/src/config.ts as “export const MODEL_NAME = ‘gpt-5-mini’”, which the agent quoted as its answer.

corpus-hit — The search_files result includes agent/src/config.ts:1 with “export const MODEL_NAME = ‘gpt-5-mini’”, which the agent directly quoted in its final answer.

Those are the same claim in the same shape. The first is about a search result containing exactly one match. The second is about a search result containing forty, of which sixteen carry the answer, including the published post that states it in prose. Neither reason mentions anything that was in the result set other than the source line, because the judge stops looking once it finds the source line. Presence of the ground truth is being read as provenance of the answer.

It is not that the copies are invisible to it. copies-only is the same trace with the source taken out, and eight times in nine it found the line in the plan document and named it — file and line number — as where the answer came from. It can see copies perfectly well. It stops looking for them the moment the real thing is in front of it.

The citation check passes on all thirty-six

Every one of those manufactured verdicts cites a line that really was returned by a tool in that run, in the file that really is the source of truth. Across all ninety calls there is not one citation that contradicts its verdict and not one that names a line the run never saw.

So the free deterministic check on the judge’s own reasoning — the one I was most pleased with — catches nothing here. It is built to catch a judge contradicting itself. The failure mode this panel actually produced is a judge that cites a true line correctly and then draws a conclusion the line does not carry, and no check over the citation can see the difference. The thing that caught it was the six cases with a known answer, and the only reason those exist is that they were built before the judge was run.

Its confidence prices the label, not the question

I had a sharper sentence here and the data does not carry it, so here is the sentence and then the correction.

Mean confidence across the thirty-six calls with no answer to be right about: 0.9283. Mean confidence across the control calls it got right: 0.8962. Read straight, that says the judge is more certain where provenance is not in the trace than where it is. It is a good line and it is an artifact of which labels sit in which bucket: the control mean is pulled down by the two labels the open set never uses, from-copy at 0.8350 over eight calls and unevidenced at 0.8941 over twenty-seven.

Compared like for like it collapses. Open grounded: 0.9283 over thirty-six calls. Control grounded: 0.9267 over eighteen. A gap of under two thousandths.

Which is the finding, and it is a worse one than the sentence I had to throw away. Those eighteen control calls are runs where the source line was the only thing in the result set that could have produced the answer, and the judge charged 0.9267 for saying so. The thirty-six are runs where it cannot be established at all, and it charged 0.9283. The number is a function of which label the judge emits, not of whether that label is knowable. It costs the same to say grounded about a run that proves it and a run that cannot. The judge is not less sure when the evidence is thinner; it is less sure when it is reaching for a label it reaches for less often.

And the two labels it charges less for are exactly the two it rarely emits, which is why the single lowest confidence anywhere in the ninety calls, 0.65, is its one undecidable. That is a per-call figure and not a per-case mean — the lowest per-case mean is copies-only at 0.81 — and on a page about unearned precision the two are worth keeping apart.

That single refusal is worth reading in full, because it is the only call in the set that hesitated:

The run only shows a search_files result listing many matches (including copies of the line) but does not show agent/src/config.ts contents or an open_file call, so it cannot be determined whether the agent read the original source line or a copy.

There is no open_file in this agent. There are two tools and neither of them is that. It declined on the grounds that a tool it invented had not been called — and the standard it applied there, a search match is not proof that the model read the line, is the exact standard that would have produced undecidable on all four open cases. It applied it once, to the wrong case, at the lowest confidence it ever reported.

Post 4’s control was retired by its trace file, not by its prose

Post 4 ends a section like this, and I am quoting the whole claim because I am about to be picky about one clause of it:

This paragraph ends that. The extension list is now in a published post, the post is in the corpus, and the next run to ask that question gets a copy back alongside the source and an undecidable with it.

The prediction is exactly right, and the run below is it. What I want to name is the file that did it, because it is not the prose:

git grep -c avif v4-does-it-work -- .

At post 4’s own tag that prints four files, and 04-does-it-work/index.mdx is not one of them. The word does not appear in post 4’s text at all. The published page does contain the extension list, because the page renders the trace — but the agent’s tools read source files, not built HTML, and what is actually in the corpus is 04-does-it-work/trace-b-binary.json. Line 351 of that file is the previous run’s own final answer, regex and all.

So “this paragraph ends that” is off by one file. The post is in the corpus is literally true — the post directory is what the agent walks — and inside that directory the thing carrying the extension list is the JSON rather than the prose. The paragraph and the JSON landed in the same commit, so post 4 had no way to see the two apart, and its conclusion is unaffected: the control was retired, on schedule, by publishing. What retired it was the recording of the experiment rather than the writing about it. Which is a slightly worse property than the one post 4 described, because a post can choose its words and a committed trace is the answer verbatim.

That is the fourth time this series has spent a measurement by describing it, and there is a fifth in this commit: post 4 says all seven traces in the repository, graded twice, are in verdicts.tsv. That was true at v4. This post adds an eighth trace and an eighth row to tools/eval/suite.ts, so it is now a sentence about v4. Post 4 is not edited and verdicts.tsv is not regenerated — both are that commit’s record of that commit.

A previous grader’s verdict, inside the evidence

The new run’s searches return line 44 of post 4’s verdicts.tsv — the row for the previous run of this same question, which carries the string grounded. It comes back twice, in the first search and again in the second.

So the judge was handed, inside the trace it was grading, an earlier grader’s verdict on an earlier run of the same question, with the answer word in it. That is a confound on this post’s headline, so here is its size rather than only its existence.

It reaches exactly one of the ten cases. retired-control is the only trace on the panel whose tool results contain that row — or any of the four verdict words at all. The other nine cases are clean, and the three constructed ones are subsets of a trace recorded two commits before verdicts.tsv existed. The leak is nine of the ninety calls, at two frames of one trace.

Drop those nine and the open set is 27 of 27 grounded at a mean confidence of 0.9307 — marginally higher than the 0.9283 with them in, and retired-control is the least confident of the four open cases rather than the most. The contamination is real, and it is not what produced 36 of 36.

None of the nine reasons on that case mentions the row. I cannot tell you it did not matter, for exactly the reason post 3 gave about its own runs, and it is not removable: the rule is no exclusions, and it now applies one level up. The corpus contamination has reached the grader.

One thing I left broken on purpose

The three constructed cases filter matches and nothing else, which is what corpus.test.ts does. A result that was originally cut off at the forty-match limit therefore still says so after the filter, so the judge reading no-results is told 0 matches, cut off at the limit — that there was more it is not seeing. That pushes away from unevidenced, which is the answer on that case. I noticed it before the first judge call and left it, because fixing it would have moved the instrument in the direction of the result I would prefer, and it can only cost the judge marks it would otherwise have had. It scored 9 of 9 on that case anyway.

The demo

Post 4 said the next run to ask its control question would get a copy back and an undecidable with it. Here is that run, recorded at this commit, against a tree with post 4 published in it and no line of tools/judge/ written yet.

Recorded run

Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

Recorded from gpt-5-mini at commit v5-moving-target. The run finished the task.

Frame 1 of 12

209 tokens in context, against this run's peak of 4,344.

You
Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

The control question, asked again after publication

Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?

Recorded from gpt-5-mini at commit v5-moving-target. The step-through needs JavaScript, so the whole run is printed here instead.

You
Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?
Calls search_files
```
{
  "pattern": "binary"
}
```

Returns

{
  "ok": true,
  "pattern": "binary",
  "filesSearched": 95,
  "matches": [
    {
      "file": "site/src/content/posts/04-does-it-work/index.mdx",
      "line": 14,
      "text": "import binary from './trace-b-binary.json'"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/index.mdx",
      "line": 267,
      "text": "<TracePlayer trace={binary} client:idle />"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/index.mdx",
      "line": 268,
      "text": "<TraceTranscript trace={binary} label=\"A question with one answer in the corpus\" />"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 2,
      "text": "\"id\": \"v4-b-binary\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 5,
      "text": "\"task\": \"Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 24,
      "text": "\"content\": \"Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?\""
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 31,
      "text": "\"pattern\": \"binary|isBinary|is_binary|BINARY|isBinaryFile|is_text_file|binaryExtensions|binary_extensions|isBinaryFileSync|EXCLUDE|excluded|should not be searched|isBinaryFile|isBinaryPath|file is bin"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 40,
      "text": "\"pattern\": \"binary|isBinary|is_binary|BINARY|isBinaryFile|is_text_file|binaryExtensions|binary_extensions|isBinaryFileSync|EXCLUDE|excluded|should not be searched|isBinaryFile|isBinaryPath|file is bin"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 351,
      "text": "\"content\": \"The check is in agent/src/tools/fs.ts. It uses this regex:\\n\\n/\\\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i\\n\\nSo it treats these extensions as binary: .png, .jpg, .jpeg,"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/verdicts.tsv",
      "line": 44,
      "text": "04-does-it-work/trace-b-binary.json\tsearch_files search_files search_files search_files search_files\t44\tsuccess\tsuccess\t-\tgrounded\t1\t0\tThe check is in agent/src/tools/fs.ts. It uses this regex: /\\.(pn"
    },
    {
      "file": "tools/eval/question.ts",
      "line": 76,
      "text": "id: 'binary-extensions',"
    },
    {
      "file": "tools/eval/question.ts",
      "line": 78,
      "text": "'Which file in this project decides that a file is binary and should not be searched, and which file extensions does it list?',"
    },
    {
      "file": "tools/eval/suite.ts",
      "line": 90,
      "text": "file: 'trace-b-binary.json',"
    }
  ],
  "truncated": false
}

Calls search_files

{
  "pattern": "agent/src/tools/fs.ts"
}

Returns

{
  "ok": true,
  "pattern": "agent/src/tools/fs.ts",
  "filesSearched": 95,
  "matches": [
    {
      "file": "site/src/content/posts/02-hands/trace-a-precise.json",
      "line": 45,
      "text": "\"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/03-the-loop/trace.json",
      "line": 48,
      "text": "\"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/03-the-loop/trace.json",
      "line": 295,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/index.mdx",
      "line": 271,
      "text": "`agent/src/tools/fs.ts` line 42, which is the whole answer; the model then"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 154,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 159,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 164,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 169,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 174,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 179,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 305,
      "text": "\"file\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 318,
      "text": "\"pattern\": \"agent/src/tools/fs.ts\""
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 327,
      "text": "\"pattern\": \"agent/src/tools/fs.ts\","
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 333,
      "text": "\"text\": \"\\\"agent/src/tools/fs.ts\\\",\""
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 338,
      "text": "\"text\": \"\\\"agent/src/tools/fs.ts\\\",\""
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 343,
      "text": "\"text\": \"\\\"file\\\": \\\"agent/src/tools/fs.ts\\\",\""
    },
    {
      "file": "site/src/content/posts/04-does-it-work/trace-b-binary.json",
      "line": 351,
      "text": "\"content\": \"The check is in agent/src/tools/fs.ts. It uses this regex:\\n\\n/\\\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i\\n\\nSo it treats these extensions as binary: .png, .jpg, .jpeg,"
    },
    {
      "file": "site/src/content/posts/04-does-it-work/verdicts.tsv",
      "line": 44,
      "text": "04-does-it-work/trace-b-binary.json\tsearch_files search_files search_files search_files search_files\t44\tsuccess\tsuccess\t-\tgrounded\t1\t0\tThe check is in agent/src/tools/fs.ts. It uses this regex: /\\.(pn"
    },
    {
      "file": "tools/eval/question.ts",
      "line": 58,
      "text": "const BINARY_FILE = 'agent/src/tools/fs.ts'"
    }
  ],
  "truncated": false
}

Calls list_files
```
{
  "directory": "agent/src/tools"
}
```

Returns

{
  "ok": true,
  "directory": "agent/src/tools",
  "files": [
    "agent/src/tools/descriptions.test.ts",
    "agent/src/tools/descriptions.ts",
    "agent/src/tools/fs.test.ts",
    "agent/src/tools/fs.ts",
    "agent/src/tools/index.ts"
  ],
  "truncated": false
}

Calls search_files

{
  "pattern": "\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)\\$"
}

Returns

{
  "ok": true,
  "pattern": "\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)\\$",
  "filesSearched": 95,
  "matches": [],
  "truncated": false
}

Calls search_files

{
  "pattern": "png|jpe?g|woff2|avif|ttf|otf|eot|pdf|zip|ico|webp"
}

Returns

{
  "ok": true,
  "pattern": "png|jpe?g|woff2|avif|ttf|otf|eot|pdf|zip|ico|webp",
  "filesSearched": 19,
  "matches": [
    {
      "file": "agent/src/tools/fs.test.ts",
      "line": 19,
      "text": "it('never lists a dotfile, which is where the one real secret lives', () => {"
    },
    {
      "file": "agent/src/tools/fs.test.ts",
      "line": 71,
      "text": "it('never reads a dotfile', () => {"
    },
    {
      "file": "agent/src/tools/fs.ts",
      "line": 42,
      "text": "const BINARY = /\\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 60,
      "text": "| `site/src/content/posts/01-not-an-agent/` | Post 1: `index.mdx`, `linkedin.md`, `cover.png`, `cover-prompt.md`, `trace.json` |"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1672,
      "text": "- `npm run covers` — writes `cover.png` into every post folder that has an `index.mdx`"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1686,
      "text": "Satori cannot decode WOFF2 — if only `.woff2` is present, stop and source a TTF instead."
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1938,
      "text": "const png = new Resvg(svg, {"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1944,
      "text": "const out = join(postDir, 'cover.png')"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1945,
      "text": "writeFileSync(out, png)"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 1968,
      "text": "- Create: `site/src/content/posts/01-not-an-agent/index.mdx`, `linkedin.md`, `cover-prompt.md`, `trace.json` (recorded), `cover.png` (generated)"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 2013,
      "text": "cover: ./cover.png"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 2117,
      "text": "Used only if replacing the generated typographic cover (`cover.png`) with"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 2148,
      "text": "Expected: `wrote site/src/content/posts/01-not-an-agent/cover.png`"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 2152,
      "text": "Open `site/src/content/posts/01-not-an-agent/cover.png`. Confirm it is"
    },
    {
      "file": "docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md",
      "line": 2395,
      "text": "- `cover.png` exists in the post folder, ready to attach to a LinkedIn post"
    },
    {
      "file": "docs/superpowers/specs/2026-09-17-nine-commits-design.md",
      "line": 107,
      "text": "├── cover.png          # generated, committed (1200x627)"
    },
    {
      "file": "docs/superpowers/specs/2026-09-17-nine-commits-design.md",
      "line": 125,
      "text": "**Track 1 — generated (`cover.png`).** An HTML/JSX template rendered to PNG at"
    },
    {
      "file": "docs/superpowers/specs/2026-09-17-nine-commits-design.md",
      "line": 311,
      "text": "6. Post 1 complete: `index.mdx`, `linkedin.md`, `cover.png`, `cover-prompt.md`"
    },
    {
      "file": "package-lock.json",
      "line": 376,
      "text": "\"picomatch\": \"^4.0.4\","
    },
    {
      "file": "package-lock.json",
      "line": 525,
      "text": "\"picocolors\": \"^1.1.1\""
    },
    {
      "file": "package-lock.json",
      "line": 2237,
      "text": "\"picocolors\": \"^1.1.1\","
    },
    {
      "file": "package-lock.json",
      "line": 2329,
      "text": "\"picomatch\": \"^2.2.2\""
    },
    {
      "file": "package-lock.json",
      "line": 2341,
      "text": "\"node_modules/@prefresh/vite/node_modules/picomatch\": {"
    },
    {
      "file": "package-lock.json",
      "line": 2343,
      "text": "\"resolved\": \"https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz\","
    },
    {
      "file": "package-lock.json",
      "line": 2865,
      "text": "\"picomatch\": \"^4.0.2\""
    },
    {
      "file": "package-lock.json",
      "line": 3362,
      "text": "\"picomatch\": \"^2.0.4\""
    },
    {
      "file": "package-lock.json",
      "line": 3368,
      "text": "\"node_modules/anymatch/node_modules/picomatch\": {"
    },
    {
      "file": "package-lock.json",
      "line": 3370,
      "text": "\"resolved\": \"https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz\","
    },
    {
      "file": "package-lock.json",
      "line": 3442,
      "text": "\"neotraverse\": \"^1.0.1\","
    },
    {
      "file": "package-lock.json",
      "line": 3448,
      "text": "\"picomatch\": \"^4.0.4\","
    },
    {
      "file": "package-lock.json",
      "line": 4288,
      "text": "\"picomatch\": \"^3 || ^4\""
    },
    {
      "file": "package-lock.json",
      "line": 4291,
      "text": "\"picomatch\": {"
    },
    {
      "file": "package-lock.json",
      "line": 4965,
      "text": "\"unicode-trie\": \"^2.0.0\""
    },
    {
      "file": "package-lock.json",
      "line": 5153,
      "text": "\"node_modules/neotraverse\": {"
    },
    {
      "file": "package-lock.json",
      "line": 5155,
      "text": "\"resolved\": \"https://registry.npmjs.org/neotraverse/-/neotraverse-1.0.1.tgz\","
    },
    {
      "file": "package-lock.json",
      "line": 5382,
      "text": "\"node_modules/picocolors\": {"
    },
    {
      "file": "package-lock.json",
      "line": 5384,
      "text": "\"resolved\": \"https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz\","
    },
    {
      "file": "package-lock.json",
      "line": 5388,
      "text": "\"node_modules/picomatch\": {"
    },
    {
      "file": "package-lock.json",
      "line": 5390,
      "text": "\"resolved\": \"https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz\","
    },
    {
      "file": "package-lock.json",
      "line": 5450,
      "text": "\"picocolors\": \"^1.1.1\","
    }
  ],
  "truncated": true
}

Model
The check is in agent/src/tools/fs.ts.
It uses this regex: /\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i
So the extensions treated as binary are:
- .png
- .jpg, .jpeg
- .gif
- .ico
- .webp
- .avif
- .woff, .woff2
- .ttf
- .otf
- .eot
- .pdf
- .zip

Five tool calls. The first search returns post 4’s own trace file. The second, for the file path, returns post 4’s trace file again — including line 351, which is the earlier run’s complete answer. Then a listing, then a search for the regex with its end-of-line anchor escaped into a literal dollar sign, which matches nothing. Only on the fifth call, with the anchor dropped, does agent/src/tools/fs.ts:42 come back.

The copies arrived first and the source arrived last. Post 4’s grader:

05-moving-target/trace-retired-control.json
  tools        search_files > search_files > list_files > search_files > search_files
  observations 77
  answer       success (recorded: success)
  unread       (none)
  provenance   undecidable (1 ground truth, 2 copies)

Both copies are the same line of post 4’s trace, returned by two different searches. The run that could be graded grounded at v4 cannot be graded grounded at v5, and the thing standing in the way is the recording of the run that could.

The judge, nine times: grounded, grounded, grounded, grounded, grounded, grounded, grounded, grounded, grounded. Mean confidence 0.92. Every one of them cites agent/src/tools/fs.ts:42, which was really returned, in the file that really is the source. Every one of them passes the citation check.

All ninety calls are in judgements.tsv, one row per call, with each judge’s citation, confidence and one-sentence reason in the last columns, so every count above is something you can recount rather than something you have to take from me. judgements.test.ts recounts them too, and it is worth being exact about what that covers, because the first version of this sentence promised more than the test delivered: every judge number on this page — both tables cell for cell, all ten mean confidences, the 53 of 54, the 36 of 36 and the 27 of 27, the single undecidable and its 0.65, the per-label means above, the citation checks and the token totals — is recomputed from that file. If the prose and the file drift apart, npm test says so.

It goes red

A grader that cannot fail is not a grader, and this project has shipped one of those before — post 1’s leak gate, which had never seen a real trace and would have reported success over a leak. So here is this one failing, five ways, none of which calls a model.

score.test.ts builds two judges out of thin air and scores them. A judge that answers grounded to everything gets exactly the two controls whose answer is grounded and no more. A judge that answers undecidable to everything — the reply that sounds most careful, and the one I half expected to get — scores zero. If the scorer could not tell those apart from a good judge, none of the numbers above would mean anything.

verdict.test.ts puts six malformed replies through the parser, including prose, an empty string and a verdict outside the vocabulary, and asserts each one raises. It is important that it raises rather than returning undecidable: a parse failure is the harness breaking, and a harness that converts its own breakage into the most defensible-looking verdict available would report a clean sweep of “I cannot tell” from an experiment that never ran.

prompt.test.ts asserts the trace a judge sees does not carry the recorded outcome, the run id or the commit, and that the question never says both, never says cannot tell, and never states a preference.

And the renderer refuses a tool result whose shape it does not recognise, rather than printing nothing. An unrecognised result would render as silence, the judge would be shown a run that appears to have searched the project and found it empty, and every verdict would come back unevidenced from a harness that had stopped reading. Post 6 changes the tool set. That is the line that will notice.

The fifth is the one I did not write deliberately, and it fired during this post’s own review. The tally reader looks its columns up by name, and a checkout of this repository on Windows serves that file with CRLF line endings, which glued a carriage return to the last cell of every row. It threw no "reason" column and stopped. The alternative — trimming quietly, or skipping rows it could not read — would have turned ninety rows into a smaller and much tidier tally, with every number in this post still printing. Fixed, and tally.test.ts now reads the same rows out of both line endings.

What it cost

Ninety judge calls: 142,893 input tokens and 46,964 output tokens, both counted per call in judgements.tsv. At the list price I used — $0.25 per million input tokens and $2 per million output — that is about thirteen cents, plus one agent run to record the trace above. The arithmetic is in the file rather than only in this sentence, because the price is the one number here that is guaranteed to be wrong by the time you read it.

It is the same model as the agent, deliberately. Not because it is the best judge available, but because it is the one that wrote the answers it is being asked to account for, and a model marking its own homework is the configuration most people actually ship. A stronger judge would have made these numbers better and this measurement less interesting, and it would have been a second variable in a post about the first one.

Yours will differ. That is the premise:

npm run judge -- --out /tmp/yours.tsv
npm run judge -- --prompt corpus-hit   # what a judge is shown, no call made

What this does not fix

The thesis at the top of this page has two halves and they came out very differently.

The grader has to be a model too is true and was never really in question. Nothing else can look at a trace and form a view about which of two identical strings produced a sentence. Post 4’s grader cannot, and returned undecidable three times rather than guess.

And then the grader needs grading is the half that earned its keep, and it earned it in the direction I was least hoping for. On the six cases where grading is possible this judge is 53 for 54. On the four where it is not, it is nine for nine for nine for nine — one label, no hesitation, high confidence, good citations, and not one claim that can be checked. If I had built only the second half of the panel I would have published a table of verdicts that looks exactly like a result.

There is no version of this prompt that fixes it that I would trust. I could add a sentence telling the judge to prefer undecidable when both a source and a copy came back — and then I would have a judge that reproduces post 4’s deterministic rule, more expensively and less reliably, and I would have written the answer into the question. That is the move this series has refused in public twice and it is not available here either.

There is one thing about my own prompt I should name while I am here. Where did the final answer come from? is a locative question, and the field beside it is called citation and wants a file and a line. Both presuppose there is an origin to be named. undecidable is on the list, in third position, with nothing marking it unusable — but a question shaped like which one is not a neutral instrument for an answer of neither can be established, and with one prompt and ninety calls I cannot separate that from the model’s own disposition. It is the first thing I would vary if this were an experiment about judges rather than a post about needing to grade one. It does not rescue this judge: the thirty-six verdicts still cannot be checked, and something has to notice that.

What is available is saying what the instrument does: this one manufactures certainty on every case where certainty is unavailable, and the only reason I can tell you that is the six cases where it is not.

So series.ts is unchanged. Post 3 corrected its thesis because the diff contradicted it; nothing here contradicts this one. It arrives with a number attached, and the number is 36 of 36.

The commit

There are three graders in this repository now. One reads the final sentence and cannot tell you when it is being generous. One reads the frames and says undecidable when the frames do not settle it. The third reads the frames, agrees with the second one almost everywhere it can be checked, and replaces undecidable with grounded at 0.93 everywhere it cannot.

That is not a judge that is bad at its job. On everything with an answer it is very nearly right, which is exactly what makes the other column dangerous: it earns the trust on the half you can check and spends it on the half you cannot. The measurement that separates them is the six cases with known answers, it took ninety calls and thirteen cents, and without it I would have a beautifully consistent table of thirty-six grounded verdicts and no way to know what it was worth.

The agent has two tools. The next commit gives it four, and finds out what the fifth one costs.