Passed Once Is Not Reliable: ThinkingBox Grades Agents on the Database, Not the Demo

A support agent handles a late delivery. It makes nine well-formed tool calls, reads the policy, updates the record, and closes with a confident summary. One field is wrong. The ticket should have stayed on hold while a carrier exception was open, and the agent marked it resolved.

That case opens the Hugging Face write-up of ThinkingBox, a sandbox and benchmark from Microsoft researchers and collaborators. The design choice matters more than any single score: grade what changed in the backend, not what the agent said it changed.

What ThinkingBox actually tests

ThinkingBox-Bench contains 507 policy-conditioned workflows across five domains: retail, hospitality and booking, auto insurance, neobank internal IT, and consulting IT and HR support. Each task starts from a known backend state. The agent gets MCP-compatible tools, a domain policy, and a user goal. A simulated user holds extra facts, such as a reference number or a date of birth, and releases them only when the agent asks.

At the end, executable checks compare the terminal state with the required state. A run fails for a wrong value, a missing change, or an extra change the task never asked for. A small subset of tasks also checks the final response with a rubric. Most do not need to. The database is the test.

Each task was run 20 times from a clean start. That repetition is the point. A single pass shows capability. Twenty passes show reliability.

Finding 1: Capability and reliability are different numbers

Claude Opus 5 scored 66.50% on a single attempt (pass@1) in the paper. Claude Opus 5.5 led the October release at 67.16% pass@1. Those are respectable numbers for multi-step work with policy constraints.

The repeat numbers fall hard. Opus 5 and Opus 5.5 each passed all 20 attempts on 241 of 507 tasks, which is 47.53%. In other words, the best agents in this test were dependable on fewer than half the workflows.

The gap is wider for models with broad coverage. Kimi-K3 solved 476 of 507 tasks at least once, the widest coverage reported, but passed all 20 attempts on only 68 tasks (13.41%). The paper reports a related pass^20 estimate of 17.60% for Kimi-K3, down from 57.37% pass@1. The two repeat figures use different estimators, so do not treat them as the same statistic. Both point the same way: solving a task once is weak evidence that an agent will solve it next Tuesday.

Model	Pass@1	Passed all 20 runs
Claude Opus 5.5	67.16%	241 of 507 tasks (47.53%)
Claude Opus 5	66.50%	241 of 507 tasks (47.53%)
Kimi-K3	57.37% (paper)	68 of 507 tasks (13.41%, October count)

Finding 2: Most failures are silent

The October ablation covered 121,680 valid trials across 12 models. Of those, 79,853 attempts failed the executable checks, about two in three trials.

The uncomfortable part is how those failures looked from the outside. 67.24% of failed trials terminated cleanly, used a state-changing tool, and reported no final tool error. Inside those quiet failures, the checks found wrong field values in 77.61%, unintended extra effects in 43.30%, and missing required effects in 25.36%. The categories overlap, so they do not add to 100%.

If your production monitoring asks whether the agent finished without an exception, this result says that signal is blind to most failures. The agent can be wrong, polite, and fully logged, and still wrong.

Finding 3: Tool handling, not reasoning, drives most failures

The Hugging Face write-up attributes roughly four in five failures to tool handling: failing to recover from an error, mishandling an empty lookup, or using a tool in the wrong order or with the wrong arguments. That attribution comes from the authors' failure analysis, not from an independent audit, and the fixes they suggest have not been measured on this benchmark yet.

Even with that caveat, the direction matches production experience. Agents rarely fail because they cannot write a paragraph about the task. They fail at the seams: a search returns nothing, an ID is in the wrong format, a policy exception applies, a second write overwrites the first. Sharpening the tool surface, adding read-back checks, and handling empty results explicitly will usually buy more reliability than swapping to a slightly larger model.

Cost makes the same point. The authors estimate cost per dependable task, meaning the cost of the full 20-run campaign divided by the tasks a model passed every time. On that measure Opus 5.5 came in at $7.80 per dependable task and Opus 5 at $13.30, for the same 241 tasks. A cheap pass@1 can become expensive once you price consistency. These are list-price estimates from the authors, not invoices.

What this means for data and AI practitioners

1. Grade state, not transcripts. For any workflow that writes to a system, write an executable check on the final record. Did the refund land, for the right amount, on the right order, with no extra changes?
2. Report a repeat metric. Add a pass-all-k or equivalent consistency number next to pass@1 in every eval report. A model choice made on pass@1 alone will mislead you.
3. Count silent failures as their own class. Track runs that ended cleanly but failed the state check. That number is your real production risk.
4. Test the ugly paths on purpose. Empty lookups, stale IDs, policy exceptions, and partial writes should be standard cases, not surprises found in production.
5. Put a read-back after every consequential write. Have the agent, or a separate checker, re-read the record and compare it with the goal before it tells the user it is done.
6. Price dependability, not demos. Compare models on cost per task passed every time, or at least on cost per verified success, not cost per attempt.

Against and in favour

Against

The benchmark is synthetic and vendor-authored. Simulated users behave more cleanly than real customers, policies are written down in full, and the backend is tidy. Real work is messier, which could make live reliability worse, not better. The October numbers also mix estimators, as the Kimi-K3 repeat figures show, so careful reading matters. And a state check can be too strict: a harmless extra note in a record is not the same as a wrong refund, even if both fail a check.

In favour

The method is open, the checks are executable, and the tasks require the things real workflows require: asking for missing facts, following a policy, sequencing dependent tools, and avoiding collateral changes. Grading terminal state is also trajectory-agnostic, so an agent can take a different valid path and still pass. That is the right bias for an eval. It rewards outcomes and stays neutral on style.

What would make me wrong

Three results would weaken this reading. First, an independent replication on live, non-synthetic workflows showing repeat reliability close to pass@1. Second, evidence that most silent failures in ThinkingBox are trivial extra effects with no customer impact, once failures are weighted by severity. Third, a harness or tool-design change that closes most of the pass@1 to all-20 gap without a model change, which would show the gap was mostly a benchmark artifact. I would welcome any of the three. None exists in the October release.

Sources

• Li et al., One Success Isn't Reliability: ThinkingBox, arXiv:2608.19741
• Microsoft ThinkingBox code release on GitHub
• ThinkingBox coverage summarising the October Hugging Face release, explainx.ai

Related on Everyday Data Science: The 100-Line Eval Harness That Catches Agent Regressions, Nous Hermes Index: agent cost per task, and MCP explained for data people.

When your agent tells a user a task is done, what do you check before you believe it: the transcript, the tool log, or the record itself?