Abstract
Agent harnesses decide whether a tool agent’s attempt is correct by asking a model. Yet Claude Opus 5.5 judges its own agent’s failures poorly, accepting 65 of the 66 attempts at AppWorld tasks the agent always fails. We introduce end-state agreement, a check that calls no model and flags an attempt whose final state is not the most common among other attempts at the task. On held-out τ2-bench customer-service tasks, agreement with four other attempts flags 48.5% of Claude Opus 5.5’s failures, against 30.9% for Claude Opus 5.5 as judge, shown the same four attempts. This preregistered comparison on identical evidence makes agreement the first check shown to beat a frontier model judging its own agent. We add compiled checks, Python functions that test the end state without a model; Claude Opus 5.5 writes them from past graded attempts. We replay AppWorld attempts in order; after the first 200 failures, the checks flag 90.1% of new failure states, those no earlier attempt reached. A judge adapted from ACE that learns lessons from the same grades flags just over half of them at the checks’ false-alarm rate, using twenty times the tokens spent writing the checks. Checks that stop learning flag fewer of these states than checks that keep learning, so a harness needs to keep revising its checks.
Code and data
verification-capital-appworld.zip ZIP · 9.4 MB · 17 files
Encrypted per the AppWorld data license, which asks that its data and derivatives not be redistributed in plain text; password: appworld-protected.
sha25688890435e87dbe3093b5e621060f3bedf5652d3c521286f0f6a81976dbefbf9dChecks
companion/capital-deploy/runs/dep/probes.jsonl.gzin verification-capital-appworld.zip350 KBCompiled checks, pass/fail run: all 261 Python checks the writer proposed (learning and anticipation arms)sha256efd6fa98abc4f19fa37d263ad94b421e01d6e8392e86c49810b8e4a2a81e70d7companion/capital-deploy/runs/dep/gates.jsonlin verification-capital-appworld.zip105 KBAdmission-gate decision for every proposed check, pass/fail run (266 admitted of 307)sha2560d1d589d0de6907d19a03433d268e5def1b8e3dc566edc5aaf76b969668a1776companion/capital-deploy/runs/dep/windows.jsonlin verification-capital-appworld.zip8.8 KBPer-window counts for the 38 windows: attempts, failures, admissions, library sizesha2568d4cf0a661ec5b4057dab0de819c361d40d31195ac3e62248483b914ea606786companion/capital-ct/results/probes.jsonl.gzin verification-capital-appworld.zip215 KBCompiled checks, graded run: learning and no-Opus arms (192 checks)sha25667b58ab04f6af0d2be40eec20fffb0ad79fcc8f04f806793659e7d75933067f2companion/capital-ct/results/gates.jsonlin verification-capital-appworld.zip87 KBAdmission-gate decisions, graded run (211 admitted of 261)sha25612bc91fa6b8dcaeddd0cc5e14083b564cc7720051a157632163fbf25d4087e37companion/capital-ct/results/anticipate/probes.jsonl.gzin verification-capital-appworld.zip213 KBCompiled checks, graded-run anticipation study (178 checks)sha2562c908ee1b57fc3ea89e110be2a3a24b9b8d9559764357e716908426a0b344b99companion/capital-ct/results/anticipate/gates.jsonlin verification-capital-appworld.zip68 KBAdmission-gate decisions, anticipation studysha2560575960abf34883dd806c65e61cbc82d20cc0da5dbbf0b9f17716278c48f591ecompanion/capital/stage0/runs/probes_final.jsonlin verification-capital-appworld.zip217 KBFrozen checks for 46 templates, instance-transfer studysha2566891c72e45ab79fe7d8d4e8be33765451c1ad1e93485155d45e10828e12bd68acompanion/capital-deploy/probe_exec.pyCheck runtime: the `ctx` object every check receives (identical in both runs)5.1 KBcompanion/capital-deploy/runs/rival_ace/ace_state.json.gzin verification-capital-appworld.zip307 KBACE-style verification playbook after the last window: the 623 bullets it learned, with their counterssha2567004ae2ffe99c31e91aa7e3e981a875ab47ea47abebddd9af14c17e245a3c009
Prompts
companion/capital-tau2/judge.pyτ²-bench zero-shot judge: system prompt and prompt builder4.2 KBcompanion/capital-tau2/render.pyτ²-bench evidence renderer used by both judges6.7 KBcompanion/tau2-agreement/peer_judge.pyPeer-judge note shown with the four peer attempts4.8 KBcompanion/tau2-agreement/tau2test/h4_judges.pyHeld-out τ²-bench judge runner that assembles both prompts3.6 KB
Verdicts
τ²-bench is © 2025 Sierra Research and used under the MIT License (https://github.com/sierra-research/tau2-bench); the τ²-bench files on this page quote and paraphrase its synthetic tasks, policies and customer records.
companion/tau2-agreement/runs/h4_zeroshot.jsonlClaude Opus 5.5 zero-shot judge verdicts on the 600 held-out τ²-bench attempts (699 records; the first verdict per attempt counts)262 KBcompanion/tau2-agreement/runs/h4_peer.jsonlClaude Opus 5.5 peer-judge verdicts on the 600 held-out τ²-bench attempts, with the four peer ids (923 records; the first verdict per attempt counts)486 KBcompanion/tau2-agreement/results/h4_agreement_verdicts.jsonlEnd-state agreement verdicts on the 600 held-out τ²-bench attempts, beside both judges' flags and the official label203 KBcompanion/tau2-agreement/results/h2_tau2_rows.jsonl.gzτ²-bench training record: agreement and zero-shot judge flags per attempt (1,133)36 KBcompanion/capital-deploy/runs/dep/verdicts.jsonl.gzin verification-capital-appworld.zip3.2 MBCompiled-check verdicts per AppWorld attempt, pass/fail run: learning, anticipation and four frozen copiessha256f9298b0f78e0fda4da83917e9435f7b19bf794167e4266c8ffbb277973583a6fcompanion/capital-deploy/eval_set.jsonlin verification-capital-appworld.zip951 KBAppWorld evaluation set: never-seen end states by windowsha256faed635dbf0ade3d105de6797de34c48bfa440ed999069f5e7d9c67d3352966ccompanion/capital-deploy/runs/labels.jsonl.gzin verification-capital-appworld.zip953 KBAppWorld official outcome and actor per attemptsha2568f7d3bec59cbec595fca9bd81c463b674a02b0f61f9f9f2cbdb23802d65ed2aecompanion/capital-deploy/runs/rival_prec/results.jsonlin verification-capital-appworld.zip396 KBPrecedent-judge verdicts with success probability and reason: every item of windows 29–38 and every never-seen correct state of windows 2–38 (1,184 items)sha256f0866600656512c58377416d8900455ca1868fe42dd434a10f9af42c9e0ddc55companion/capital-deploy/runs/rival_ace/results.jsonlin verification-capital-appworld.zip1.1 MBACE-style playbook verdicts with success probability, reason and bullets used (2,749 items)sha256ced147d2655eb80f9752bcf8b6b1e3f92702eb2d3c30cde8dd1c4ae64e115f07companion/capital-ct/results/verdicts.jsonl.gzin verification-capital-appworld.zip2.9 MBCompiled-check verdicts, graded run: learning, no-Opus and four frozen copiessha256838b571ab27ec444970da91fdec2e3dc66cb0eac17920d754ac1f2a88126cb10companion/capital-ct/results/anticipate/verdicts.jsonl.gzin verification-capital-appworld.zip823 KBCompiled-check verdicts, anticipation studysha2568f844183748128e7e829ae4695bdfc893895a42019fdb16ca27b339d0f4df15a
AppWorld notice
APPWORLD_CANARY.txtin verification-capital-appworld.zip457 BAppWorld canary string, as AppWorld's README gives itsha25635af9554df01420d2abf57ba29f65015babf4ef865f719bcf4a1eadf5cfc5695
Cite
@misc{cumes2026zerocall,
title = {{Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on $\tau^2$-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld}},
author = {Cumes, Lindell S.},
year = {2026},
url = {https://durableintel.com/paper-verification-capital}
}