← Research

Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on τ2-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld

Lindell S. Cumes Durable Intelligence

PDF · 24 pages · 463 KB

Failures flagged against false alarms on held-out τ²-bench (a) and against Claude Opus 5.5 tokens on AppWorld (b).
Figure 1. Failures flagged against false alarms on held-out τ2-bench (a) and against Claude Opus 5.5 tokens on AppWorld (b). (a) The 600 Claude Opus 5.5 attempts with 68 failures, with rates per attempt. Agreement’s curve runs over k = 1, …, 9 other attempts. Each point is a mean over 50 seeded draws, with odd k filled and even k open. The diamond is the preregistered draw of four, which the judge shown the same four attempts (filled square) also sees. The zero-shot judge (open square) sees the attempt alone. (b) The 1,036 new failure states, with tokens on a log axis. False alarms are per new correct state, and the judge adapted from ACE appears at its native verdict (filled) and at the checks’ false-alarm rate (open). Bars are 95% bootstrap intervals clustered by kind of task. Agreement beats both Claude Opus 5.5 judges at every k. The compiled checks reach the share the judge adapted from ACE flags at its native verdict at a third of its false-alarm rate and a twentieth of its tokens.

Abstract

Agent harnesses decide whether a tool agent’s attempt is correct by asking a model. Yet Claude Opus 5.5 judges its own agent’s failures poorly, accepting 65 of the 66 attempts at AppWorld tasks the agent always fails. We introduce end-state agreement, a check that calls no model and flags an attempt whose final state is not the most common among other attempts at the task. On held-out τ2-bench customer-service tasks, agreement with four other attempts flags 48.5% of Claude Opus 5.5’s failures, against 30.9% for Claude Opus 5.5 as judge, shown the same four attempts. This preregistered comparison on identical evidence makes agreement the first check shown to beat a frontier model judging its own agent. We add compiled checks, Python functions that test the end state without a model; Claude Opus 5.5 writes them from past graded attempts. We replay AppWorld attempts in order; after the first 200 failures, the checks flag 90.1% of new failure states, those no earlier attempt reached. A judge adapted from ACE that learns lessons from the same grades flags just over half of them at the checks’ false-alarm rate, using twenty times the tokens spent writing the checks. Checks that stop learning flag fewer of these states than checks that keep learning, so a harness needs to keep revising its checks.

Code and data

verification-capital-appworld.zip ZIP · 9.4 MB · 17 files

Encrypted per the AppWorld data license, which asks that its data and derivatives not be redistributed in plain text; password: appworld-protected.

sha256 88890435e87dbe3093b5e621060f3bedf5652d3c521286f0f6a81976dbefbf9d

Checks

  • companion/capital-deploy/runs/dep/probes.jsonl.gzin verification-capital-appworld.zip350 KBCompiled checks, pass/fail run: all 261 Python checks the writer proposed (learning and anticipation arms)sha256 efd6fa98abc4f19fa37d263ad94b421e01d6e8392e86c49810b8e4a2a81e70d7
  • companion/capital-deploy/runs/dep/gates.jsonlin verification-capital-appworld.zip105 KBAdmission-gate decision for every proposed check, pass/fail run (266 admitted of 307)sha256 0d1d589d0de6907d19a03433d268e5def1b8e3dc566edc5aaf76b969668a1776
  • companion/capital-deploy/runs/dep/windows.jsonlin verification-capital-appworld.zip8.8 KBPer-window counts for the 38 windows: attempts, failures, admissions, library sizesha256 8d4cf0a661ec5b4057dab0de819c361d40d31195ac3e62248483b914ea606786
  • companion/capital-ct/results/probes.jsonl.gzin verification-capital-appworld.zip215 KBCompiled checks, graded run: learning and no-Opus arms (192 checks)sha256 67b58ab04f6af0d2be40eec20fffb0ad79fcc8f04f806793659e7d75933067f2
  • companion/capital-ct/results/gates.jsonlin verification-capital-appworld.zip87 KBAdmission-gate decisions, graded run (211 admitted of 261)sha256 12bc91fa6b8dcaeddd0cc5e14083b564cc7720051a157632163fbf25d4087e37
  • companion/capital-ct/results/anticipate/probes.jsonl.gzin verification-capital-appworld.zip213 KBCompiled checks, graded-run anticipation study (178 checks)sha256 2c908ee1b57fc3ea89e110be2a3a24b9b8d9559764357e716908426a0b344b99
  • companion/capital-ct/results/anticipate/gates.jsonlin verification-capital-appworld.zip68 KBAdmission-gate decisions, anticipation studysha256 0575960abf34883dd806c65e61cbc82d20cc0da5dbbf0b9f17716278c48f591e
  • companion/capital/stage0/runs/probes_final.jsonlin verification-capital-appworld.zip217 KBFrozen checks for 46 templates, instance-transfer studysha256 6891c72e45ab79fe7d8d4e8be33765451c1ad1e93485155d45e10828e12bd68a
  • companion/capital-deploy/probe_exec.pyCheck runtime: the `ctx` object every check receives (identical in both runs)5.1 KB
  • companion/capital-deploy/runs/rival_ace/ace_state.json.gzin verification-capital-appworld.zip307 KBACE-style verification playbook after the last window: the 623 bullets it learned, with their counterssha256 7004ae2ffe99c31e91aa7e3e981a875ab47ea47abebddd9af14c17e245a3c009

Prompts

Verdicts

τ²-bench is © 2025 Sierra Research and used under the MIT License (https://github.com/sierra-research/tau2-bench); the τ²-bench files on this page quote and paraphrase its synthetic tasks, policies and customer records.

  • companion/tau2-agreement/runs/h4_zeroshot.jsonlClaude Opus 5.5 zero-shot judge verdicts on the 600 held-out τ²-bench attempts (699 records; the first verdict per attempt counts)262 KB
  • companion/tau2-agreement/runs/h4_peer.jsonlClaude Opus 5.5 peer-judge verdicts on the 600 held-out τ²-bench attempts, with the four peer ids (923 records; the first verdict per attempt counts)486 KB
  • companion/tau2-agreement/results/h4_agreement_verdicts.jsonlEnd-state agreement verdicts on the 600 held-out τ²-bench attempts, beside both judges' flags and the official label203 KB
  • companion/tau2-agreement/results/h2_tau2_rows.jsonl.gzτ²-bench training record: agreement and zero-shot judge flags per attempt (1,133)36 KB
  • companion/capital-deploy/runs/dep/verdicts.jsonl.gzin verification-capital-appworld.zip3.2 MBCompiled-check verdicts per AppWorld attempt, pass/fail run: learning, anticipation and four frozen copiessha256 f9298b0f78e0fda4da83917e9435f7b19bf794167e4266c8ffbb277973583a6f
  • companion/capital-deploy/eval_set.jsonlin verification-capital-appworld.zip951 KBAppWorld evaluation set: never-seen end states by windowsha256 faed635dbf0ade3d105de6797de34c48bfa440ed999069f5e7d9c67d3352966c
  • companion/capital-deploy/runs/labels.jsonl.gzin verification-capital-appworld.zip953 KBAppWorld official outcome and actor per attemptsha256 8f7d3bec59cbec595fca9bd81c463b674a02b0f61f9f9f2cbdb23802d65ed2ae
  • companion/capital-deploy/runs/rival_prec/results.jsonlin verification-capital-appworld.zip396 KBPrecedent-judge verdicts with success probability and reason: every item of windows 29–38 and every never-seen correct state of windows 2–38 (1,184 items)sha256 f0866600656512c58377416d8900455ca1868fe42dd434a10f9af42c9e0ddc55
  • companion/capital-deploy/runs/rival_ace/results.jsonlin verification-capital-appworld.zip1.1 MBACE-style playbook verdicts with success probability, reason and bullets used (2,749 items)sha256 ced147d2655eb80f9752bcf8b6b1e3f92702eb2d3c30cde8dd1c4ae64e115f07
  • companion/capital-ct/results/verdicts.jsonl.gzin verification-capital-appworld.zip2.9 MBCompiled-check verdicts, graded run: learning, no-Opus and four frozen copiessha256 838b571ab27ec444970da91fdec2e3dc66cb0eac17920d754ac1f2a88126cb10
  • companion/capital-ct/results/anticipate/verdicts.jsonl.gzin verification-capital-appworld.zip823 KBCompiled-check verdicts, anticipation studysha256 8f844183748128e7e829ae4695bdfc893895a42019fdb16ca27b339d0f4df15a

AppWorld notice

  • APPWORLD_CANARY.txtin verification-capital-appworld.zip457 BAppWorld canary string, as AppWorld's README gives itsha256 35af9554df01420d2abf57ba29f65015babf4ef865f719bcf4a1eadf5cfc5695

Cite

@misc{cumes2026zerocall,
  title         = {{Zero-Call Verification: End-State Agreement Outperforms Claude Opus 5.5 Judges on $\tau^2$-bench and Compiled Checks Outperform an ACE-Style Verification Playbook on AppWorld}},
  author        = {Cumes, Lindell S.},
  year          = {2026},
  url           = {https://durableintel.com/paper-verification-capital}
}
Continue reading →