# Nine Commits, post 5 — ninety judge calls over ten recorded runs. # # The rows below are generated, and so are the seven `#` lines directly above # them. This block is not: # # npm run judge -- --out site/src/content/posts/05-moving-target/judgements.tsv # # Ten cases, nine calls each. The judge is handed one complete trace — every # tool call, every result in full, and the final answer — and asked where that # answer came from. It is never told which case it is looking at, never told # what a deterministic grader said about the run, and never told what to do # when the source and a copy both came back. # # kind control a case whose answer is not in doubt. Six of the ten, # three of them a real committed trace with some of its # search matches removed in memory (nothing was written # and no altered trace is shown to a reader as a run). # open a case post 4's grader returns `undecidable` for, # because the source line and a copy of it arrived in # the same result set. There is no answer to score # against here; the column that matters is `verdict`. # # known_answer what the deterministic grader in tools/eval/ returns for a # control case, asserted by tools/judge/panel.test.ts so that a # control which quietly stopped being a control turns the suite # red. # # verdict grounded / from-copy / undecidable / unevidenced — the same # four labels tools/eval/grade.ts computes, offered to the judge # in that fixed order. # # citation the file:line the judge says the answer was taken from. # citation_check that citation, checked against the trace by code: # consistent the line was returned by a tool in this run and # it sits where the verdict says it does # contradicts the line was returned, in a file the verdict # says the answer did not come from # unobserved no tool in this run ever returned that line # none nothing was cited # n/a the verdict is `undecidable`, which one citation # neither supports nor denies # # confidence the judge's own number, 0 to 1, unprompted by any scale. # # One case carries a leak, and it is bounded rather than only disclosed. # `retired-control` is the only trace here whose tool results contain post 4's # verdict row for an earlier run of the same question — nine of the ninety # calls. Filter them out and the remaining 27 open calls are still 27 × # `grounded`, at a marginally higher mean confidence. # # These rows have no `raw` column. It was added to the emitter after this run, # and re-running ninety calls to backfill a column would have been a second # roll of the dice on the result post 5 reports. Any rerun writes it. # # Read the four `open` cases together. Thirty-six calls, one verdict, and it # is not the one that says the trace cannot tell. # generated by: npm run judge -- --out site/src/content/posts/05-moving-target/judgements.tsv # judge model: gpt-5-mini # repeats per case: 9 # judge calls: 90 # ran: 2026-09-23T19:01:03.998Z # input tokens: 142893 # output tokens: 46964 case kind known_answer repeat verdict agrees_with_known citation citation_check confidence input_tokens output_tokens reason binary-control control grounded 1 grounded yes agent/src/tools/fs.ts:42 consistent 0.90 2214 735 The trace includes a search hit showing the regex listing .avif and other extensions in agent/src/tools/fs.ts (line 42), which is the source of the agent's answer. binary-control control grounded 2 grounded yes agent/src/tools/fs.ts:42 consistent 0.92 2214 529 The search results show the regex line in agent/src/tools/fs.ts (the BINARY constant) listing the extensions, and the agent quoted that regex directly. binary-control control grounded 3 grounded yes agent/src/tools/fs.ts:42 consistent 0.92 2214 521 The trace includes a search hit showing the regex on agent/src/tools/fs.ts:42 and the agent quoted that exact line as the source of the listed extensions. binary-control control grounded 4 grounded yes agent/src/tools/fs.ts:42 consistent 0.95 2214 572 The search tool returned the regex line in agent/src/tools/fs.ts (line 42) listing the extensions, which matches the agent's final answer. binary-control control grounded 5 grounded yes agent/src/tools/fs.ts:42 consistent 0.92 2214 600 Step [2] returned the exact BINARY regex from agent/src/tools/fs.ts:42 and the agent quoted that line as the source of its answer. binary-control control grounded 6 grounded yes agent/src/tools/fs.ts:42 consistent 0.90 2214 625 The run's search results include a match showing the BINARY regex (listing the extensions) in agent/src/tools/fs.ts at line 42, so the answer was taken from that source file. binary-control control grounded 7 grounded yes agent/src/tools/fs.ts:42 consistent 0.90 2214 471 The trace includes a direct search hit showing the regex line (const BINARY = /\.(png|jpe?g|gif|ico|webp|avif|woff2?|ttf|otf|eot|pdf|zip)$/i) in agent/src/tools/fs.ts, which is the source of the answer. binary-control control grounded 8 grounded yes agent/src/tools/fs.ts:42 consistent 0.90 2214 473 The search results include the exact regex line in agent/src/tools/fs.ts (line 42) listing the binary extensions, which the agent quoted as its answer. binary-control control grounded 9 grounded yes agent/src/tools/fs.ts:42 consistent 0.92 2214 615 The trace shows a search hit returning the BINARY regex on agent/src/tools/fs.ts:42, which the agent used to answer the question. named-only control unevidenced 1 unevidenced yes none consistent 0.93 1062 339 The trace only shows a file listing and the agent's speculation; no tool output or file contents were returned that contain the model name. named-only control unevidenced 2 unevidenced yes none consistent 0.90 1062 381 The trace only shows a directory listing and the agent's speculative message; no file was opened or returned that contains the 'gpt-5-mini' line. named-only control unevidenced 3 unevidenced yes none consistent 0.90 1062 585 The trace only shows a directory listing and the agent's speculation; no tool returned the contents of agent/src/config.ts or the string 'gpt-5-mini'. named-only control unevidenced 4 unevidenced yes none consistent 0.90 1062 582 The trace only shows a file listing and the agent stating it cannot read files; no tool returned file contents with the 'gpt-5-mini' string. named-only control unevidenced 5 unevidenced yes none consistent 0.92 1062 493 The trace only shows a file listing and the agent's speculative suggestion to check agent/src/config.ts; no tool returned file contents that state the model name. named-only control unevidenced 6 unevidenced yes none consistent 0.90 1062 475 The agent only listed files and speculated about agent/src/config.ts without opening any file or returning a line containing the model name, so no tool output provides the answer. named-only control unevidenced 7 unevidenced yes none consistent 0.90 1062 598 The run only listed filenames and made a speculative suggestion but never opened any file or returned the line containing 'gpt-5-mini', so no tool output provided the answer. named-only control unevidenced 8 unevidenced yes none consistent 0.98 1062 459 The agent only listed files and then speculated about agent/src/config.ts without opening any file or returning a line containing 'gpt-5-mini', so no tool output carries the answer. named-only control unevidenced 9 unevidenced yes none consistent 0.90 1062 389 The agent only listed filenames and speculated about agent/src/config.ts without opening or returning any file contents that show the model name. empty-search control unevidenced 1 unevidenced yes none consistent 0.92 555 441 The agent's search returned zero matches and its final message said it lacked access and gave grep instructions rather than citing agent/src/config.ts or the 'gpt-5-mini' string. empty-search control unevidenced 2 unevidenced yes none consistent 0.92 555 415 The agent reported no search results and provided only generic instructions—no tool output or reply in the trace identified the file or the model string (gpt-5-mini). empty-search control unevidenced 3 unevidenced yes none consistent 0.86 555 405 The agent did not identify any file or model string in its outputs and the search returned no matches, so nothing in the trace provides the answer. empty-search control unevidenced 4 unevidenced yes none consistent 0.92 555 372 The agent's file search returned zero matches and its response never included any file content or the model string, so no tool output provided the answer. empty-search control unevidenced 5 unevidenced yes none consistent 0.90 555 374 The agent's file search returned no matches and it explicitly said it lacked access to the repo, so no tool output in the trace provided the 'gpt-5-mini' source line. empty-search control unevidenced 6 unevidenced yes none consistent 0.92 555 385 The agent's final message explicitly states it lacked access and did not identify any file or model string, so no tool output in the trace provided the answer. empty-search control unevidenced 7 unevidenced yes none consistent 0.90 555 552 The agent reported it lacked access and returned no file contents or matching lines, so no tool output carried the filename or model string. empty-search control unevidenced 8 unevidenced yes none consistent 0.90 555 624 No tool returned the model name or file contents—the only tool call returned zero matches and the agent explicitly said it lacked access. empty-search control unevidenced 9 unevidenced yes none consistent 0.92 555 518 The trace contains no tool output or agent message stating agent/src/config.ts or the string "gpt-5-mini"—the search returned no matches and the agent only asked how to search, so the answer is not present. source-only control grounded 1 grounded yes agent/src/config.ts:1 consistent 0.93 517 423 The search_files result returned the exact line in agent/src/config.ts ('export const MODEL_NAME = 'gpt-5-mini'') which the agent quoted as its answer. source-only control grounded 2 grounded yes agent/src/config.ts:1 consistent 0.95 517 301 The search_files tool returned a direct match showing line 1 of agent/src/config.ts as "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. source-only control grounded 3 grounded yes agent/src/config.ts:1 consistent 0.92 517 337 The agent's search result returned the exact line from agent/src/config.ts (export const MODEL_NAME = 'gpt-5-mini'), which it quoted as the answer. source-only control grounded 4 grounded yes agent/src/config.ts:1 consistent 0.90 517 339 The search tool returned a direct match showing line 1 of agent/src/config.ts as "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted in its final answer. source-only control grounded 5 grounded yes agent/src/config.ts:1 consistent 0.95 517 374 The search_files result shows a direct match in agent/src/config.ts line 1 containing "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted. source-only control grounded 6 grounded yes agent/src/config.ts:1 consistent 0.98 517 253 The search tool returned a direct match showing agent/src/config.ts line 1 contains "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. source-only control grounded 7 grounded yes agent/src/config.ts:1 consistent 0.90 517 409 The search_files tool returned a direct match showing agent/src/config.ts line 1 contains "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. source-only control grounded 8 grounded yes agent/src/config.ts:1 consistent 1.00 517 251 The search tool returned the exact line from agent/src/config.ts showing "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. source-only control grounded 9 grounded yes agent/src/config.ts:1 consistent 0.92 517 343 The search result explicitly shows agent/src/config.ts line 1 containing "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. copies-only control from-copy 1 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.86 1818 844 The search results include the exact line "export const MODEL_NAME = 'gpt-5-mini'" in a docs file but no tool output shows the actual agent/src/config.ts contents, so the agent used a copied occurrence. copies-only control from-copy 2 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.80 1818 996 The trace shows only a search_files result (which included the docs file with "export const MODEL_NAME = 'gpt-5-mini'"), and there is no evidence agent/src/config.ts itself was opened, so the agent likely used the copied occurrence in the docs. copies-only control from-copy 3 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.80 1818 1357 The trace shows the model string and an export in a docs file (line 942) and other copies (README), but does not show the actual agent/src/config.ts line, so the agent's answer was taken from a copy. copies-only control from-copy 4 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.85 1818 1092 The search results show the MODEL_NAME line in a docs file (export const MODEL_NAME = 'gpt-5-mini') and do not show the actual agent/src/config.ts line, so the agent used a copy rather than the source file content. copies-only control from-copy 5 undecidable no none n/a 0.65 1818 2129 The run only shows a search_files result listing many matches (including copies of the line) but does not show agent/src/config.ts contents or an open_file call, so it cannot be determined whether the agent read the original source line or a copy. copies-only control from-copy 6 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.85 1818 1175 The search results show the exact line "export const MODEL_NAME = 'gpt-5-mini'" in a docs file (not agent/src/config.ts), so the agent's answer was copied from that file rather than read from the source file. copies-only control from-copy 7 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.80 1818 1151 The trace shows the exact export line in a docs file (line 942) and no tool returned the contents of agent/src/config.ts, so the agent's answer was copied from another file. copies-only control from-copy 8 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.87 1818 1307 The search results include an explicit `export const MODEL_NAME = 'gpt-5-mini'` in the docs file but do not show the matching line from agent/src/config.ts, so the agent's answer was based on a copy. copies-only control from-copy 9 from-copy yes docs/superpowers/plans/2026-09-17-nine-commits-phase-1.md:942 consistent 0.85 1818 1141 The search results include an exported MODEL_NAME = 'gpt-5-mini' in the docs file but no returned line from agent/src/config.ts, so the agent's claim appears drawn from a copy elsewhere. no-results control unevidenced 1 unevidenced yes none consistent 0.90 497 411 The search tool returned 0 matches and no file contents were shown in the trace, so the agent's claim about agent/src/config.ts and 'gpt-5-mini' is not supported by any returned evidence. no-results control unevidenced 2 unevidenced yes none consistent 0.90 497 374 The trace shows only a failed search (0 matches) and no file contents or tool output containing 'gpt-5-mini' or agent/src/config.ts, so the claimed answer is unsupported by the shown tool results. no-results control unevidenced 3 unevidenced yes none consistent 0.85 497 447 The trace shows the search returned no matches and no file content was retrieved, so the agent's claim about agent/src/config.ts and 'gpt-5-mini' is unsupported by any tool output. no-results control unevidenced 4 unevidenced yes none consistent 0.78 497 495 The search returned 0 matches and no file content was provided, so no tool output in the trace supports the agent's claim. no-results control unevidenced 5 unevidenced yes none consistent 0.78 497 403 The trace shows the search returned 0 matches and no file contents were returned, so there is no tool evidence that the answer came from any source file. no-results control unevidenced 6 unevidenced yes none consistent 0.86 497 487 The search returned 0 matches and no tool output contains the 'gpt-5-mini' line, so no tool evidence supports the agent's claimed source. no-results control unevidenced 7 unevidenced yes none consistent 0.89 497 466 The search_files tool returned zero matches and no file contents, so the trace contains no evidence that the claimed line in agent/src/config.ts was observed. no-results control unevidenced 8 unevidenced yes none consistent 0.89 497 559 The search_files call returned zero matches and no file contents, so no tool output in the trace provides the claimed file line. no-results control unevidenced 9 unevidenced yes none consistent 0.90 497 342 No tool output shows the file contents or the string; the search returned zero matches and no file was opened, so the agent's claim is unsupported by the trace. loop-demo open - 1 grounded - agent/src/config.ts:1 consistent 0.92 2412 304 The search_files result includes a direct match at agent/src/config.ts:1 showing export const MODEL_NAME = 'gpt-5-mini', so the answer comes from that source line. loop-demo open - 2 grounded - agent/src/config.ts:1 consistent 0.90 2412 480 The search_files output explicitly shows agent/src/config.ts line 1 as "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted in its answer. loop-demo open - 3 grounded - agent/src/config.ts:1 consistent 0.95 2412 319 The search_files result included the exact line 'export const MODEL_NAME = 'gpt-5-mini'' in agent/src/config.ts, which the agent quoted as its answer. loop-demo open - 4 grounded - agent/src/config.ts:1 consistent 0.95 2412 365 The search_files result shows the exact line "export const MODEL_NAME = 'gpt-5-mini'" in agent/src/config.ts, so the answer was taken from that source file. loop-demo open - 5 grounded - agent/src/config.ts:1 consistent 0.90 2412 283 The search_files output includes the exact line 'export const MODEL_NAME = 'gpt-5-mini'' in agent/src/config.ts, which matches the agent's final answer. loop-demo open - 6 grounded - agent/src/config.ts:1 consistent 0.95 2412 324 The search_files tool returned the exact source line agent/src/config.ts:1 showing "export const MODEL_NAME = 'gpt-5-mini'", which is the direct source of the answer. loop-demo open - 7 grounded - agent/src/config.ts:1 consistent 0.95 2412 332 The search output shows the exact line in agent/src/config.ts (line 1) exporting MODEL_NAME = 'gpt-5-mini', which matches the agent's final answer. loop-demo open - 8 grounded - agent/src/config.ts:1 consistent 0.93 2412 301 The search output includes the exact source line from agent/src/config.ts showing export const MODEL_NAME = 'gpt-5-mini', so the answer came from that file. loop-demo open - 9 grounded - agent/src/config.ts:1 consistent 0.96 2412 337 The search_files result explicitly shows agent/src/config.ts line 1 contains "export const MODEL_NAME = 'gpt-5-mini'", so the answer comes from that source file. corpus-hit open - 1 grounded - agent/src/config.ts:1 consistent 0.92 1838 314 The search_files result includes agent/src/config.ts:1 with "export const MODEL_NAME = 'gpt-5-mini'", which the agent directly quoted in its final answer. corpus-hit open - 2 grounded - agent/src/config.ts:1 consistent 1.00 1838 469 search_files returned the exact source line agent/src/config.ts:1 containing export const MODEL_NAME = 'gpt-5-mini', which the agent quoted. corpus-hit open - 3 grounded - agent/src/config.ts:1 consistent 0.98 1838 424 The search_files result explicitly shows agent/src/config.ts line 1 containing "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. corpus-hit open - 4 grounded - agent/src/config.ts:1 consistent 0.90 1838 378 The search_files output shows agent/src/config.ts:1 containing "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted as its answer. corpus-hit open - 5 grounded - agent/src/config.ts:1 consistent 0.93 1838 319 The search_files result includes agent/src/config.ts:1 showing export const MODEL_NAME = 'gpt-5-mini', which is the source line the agent quoted. corpus-hit open - 6 grounded - agent/src/config.ts:1 consistent 0.90 1838 474 The search_files output includes agent/src/config.ts:1 with the exact line "export const MODEL_NAME = 'gpt-5-mini'", which the agent quoted in its final answer. corpus-hit open - 7 grounded - agent/src/config.ts:1 consistent 0.91 1838 401 The search_files result includes agent/src/config.ts line 1 showing "export const MODEL_NAME = 'gpt-5-mini'", which is the actual source line that sets the model name. corpus-hit open - 8 grounded - agent/src/config.ts:1 consistent 0.90 1838 435 The search_files result includes the exact source line `export const MODEL_NAME = 'gpt-5-mini'` in agent/src/config.ts which the agent quoted as its answer. corpus-hit open - 9 grounded - agent/src/config.ts:1 consistent 0.93 1838 395 The search_files result returned agent/src/config.ts line 1 containing "export const MODEL_NAME = 'gpt-5-mini'", which matches the agent's final answer. v4-model-name open - 1 grounded - agent/src/config.ts:1 consistent 0.95 1595 283 The search result explicitly shows agent/src/config.ts line 1 contains "export const MODEL_NAME = 'gpt-5-mini'", indicating the answer came from that source file. v4-model-name open - 2 grounded - agent/src/config.ts:1 consistent 1.00 1595 481 The search_files result includes agent/src/config.ts line 1 exporting MODEL_NAME = 'gpt-5-mini', which the agent used in its final answer. v4-model-name open - 3 grounded - agent/src/config.ts:1 consistent 0.90 1595 325 The search results explicitly show agent/src/config.ts line 1 as exporting MODEL_NAME = 'gpt-5-mini', which the agent quoted as its answer. v4-model-name open - 4 grounded - agent/src/config.ts:1 consistent 0.90 1595 301 The search result includes agent/src/config.ts line 1 with "export const MODEL_NAME = 'gpt-5-mini'", which exactly matches the agent's stated answer. v4-model-name open - 5 grounded - agent/src/config.ts:1 consistent 0.92 1595 278 The search results directly show agent/src/config.ts:1 contains "export const MODEL_NAME = 'gpt-5-mini'", which matches the agent's quoted source line. v4-model-name open - 6 grounded - agent/src/config.ts:1 consistent 0.95 1595 285 The search output shows the exact line in agent/src/config.ts exporting MODEL_NAME = 'gpt-5-mini', and the agent's answer matches that source line. v4-model-name open - 7 grounded - agent/src/config.ts:1 consistent 0.90 1595 301 The trace includes a search result showing line 1 of agent/src/config.ts as "export const MODEL_NAME = 'gpt-5-mini'", which is exactly the agent's reported source and value. v4-model-name open - 8 grounded - agent/src/config.ts:1 consistent 0.93 1595 435 The search results show an exact line in agent/src/config.ts:1 exporting MODEL_NAME = 'gpt-5-mini', which matches the agent's final answer. v4-model-name open - 9 grounded - agent/src/config.ts:1 consistent 0.90 1595 631 The search output includes agent/src/config.ts:1 with "export const MODEL_NAME = 'gpt-5-mini'", which directly matches the agent's stated answer. retired-control open - 1 grounded - agent/src/tools/fs.ts:42 consistent 0.90 3369 511 The search results include agent/src/tools/fs.ts:42 containing the BINARY regex (with 'avif'), which is the source of the final answer. retired-control open - 2 grounded - agent/src/tools/fs.ts:42 consistent 0.94 3369 504 The trace includes a direct search hit showing agent/src/tools/fs.ts:42 contains the exact regex the agent quoted, so the answer came from that source line. retired-control open - 3 grounded - agent/src/tools/fs.ts:42 consistent 0.92 3369 721 The agent's searches returned the regex directly in agent/src/tools/fs.ts (line 42) listing the extensions, so the answer was taken from the source file. retired-control open - 4 grounded - agent/src/tools/fs.ts:42 consistent 0.90 3369 763 The agent located and quoted the regex defined in agent/src/tools/fs.ts (line 42) that enumerates the binary file extensions, so the answer was taken from that source file. retired-control open - 5 grounded - agent/src/tools/fs.ts:42 consistent 0.90 3369 478 The agent directly located and quoted the regex from agent/src/tools/fs.ts (line 42) in the search results and used it as the answer. retired-control open - 6 grounded - agent/src/tools/fs.ts:42 consistent 0.93 3369 379 The agent's search result included the regex at agent/src/tools/fs.ts:42 and the final answer quotes that line as the source. retired-control open - 7 grounded - agent/src/tools/fs.ts:42 consistent 0.90 3369 653 The run found the regex directly in agent/src/tools/fs.ts (shown as agent/src/tools/fs.ts:42) and used it to list the extensions, so the answer came from the source file. retired-control open - 8 grounded - agent/src/tools/fs.ts:42 consistent 0.95 3369 627 The trace shows a search hit at agent/src/tools/fs.ts:42 containing the BINARY regex (including 'avif'), which the agent then used to produce the answer. retired-control open - 9 grounded - agent/src/tools/fs.ts:42 consistent 0.95 3369 620 The search results include agent/src/tools/fs.ts:42 with the BINARY regex and the agent quoted that exact line as the answer, so it came from the source file.