Switching Agent Harnesses Changed 13% of Tasks. Rerunning the Same One Changed 13% Too

Every week a vendor announces a harness change and a pass-rate gain. A new system prompt, a new tool set, a new context manager, and a leaderboard number two or three points higher than last month. The question almost nobody asks is whether that gain is real, because almost nobody measures what the score does when nothing changes at all.

A preprint from Yangze Liu and Zhongyi Han at Shandong University, submitted to arXiv on October 3, 2026, does exactly that measurement. It is called "What Does a Harness Buy? Tokens, Mostly" (arXiv:2610.04433), and its answer is uncomfortable for anyone who has ever chosen a harness on pass rate alone: on 447 real coding tasks, the wrapper around the model barely moves the score, and the score moves plenty on its own.

What the study actually ran

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. The authors put five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, a set of real GitHub issues with tests.

Claude Code is the heavy option: a long system prompt, typed file and search tools, automatic context compaction. mini-SWE-agent is the light one: a single bash tool and almost no context management. OpenCode sits between them, with typed tools and a smaller preamble.

The benchmark pool started at 500 tasks. Eight that need external services were removed, leaving 492. The authors used a 447-task pool for two models and a 45-task hard subset for all five. Claude Opus 5 appears only inside Claude Code, as a ceiling reference.

The part that makes this study worth reading is the control. The authors ran ten pairs of runs with the same model, the same harness, the same tasks, and the same settings. That gives an estimate of how much a score moves when nothing changes but the run. Without that baseline, a two or three point harness gain is unreadable.

Finding 1: 13% flips either way

At the task level, reruns flip 13% of outcomes at the median. Harness swaps flip the same 13%. Model swaps flip 22%.

What changed	Tasks that flip outcome
Nothing, same configuration rerun	13%
Harness swapped, model fixed	13%
Model swapped, harness fixed	22%

The score changes less than the flip rate, because flips go both ways: some tasks that passed now fail, and some that failed now pass. But the churn is the point. A task a harness wins in one run is not reliably the task it wins in the next.

If you swap harnesses and your pass rate moves two points, you have learned nothing about the harness. You have learned that agent benchmarks are noisy at the task level, and that a single run is an anecdote with a leaderboard attached.

Finding 2: heavy versus light is a draw

On the 447-task pool, the heaviest and lightest harnesses, Claude Code and mini-SWE-agent, are equivalent within five points for both models tested. The measured gaps were minus 1.8 and minus 1.4 points, with 90% intervals inside plus or minus five.

Read that again slowly. The harness with the long system prompt, the typed tools, and the automatic compaction scored no better than the harness with one bash tool and almost no context management. All of that engineering, and the score cannot tell them apart.

This is not a claim that harness engineering is useless. It is a claim about what a pass-rate number can show. The harness determines failure modes and cost, and the next finding shows one case where a harness loses real points. But if your evidence for a harness is a two-point lead on a few hundred tasks, you do not have evidence yet.

Finding 3: the one real effect is a loss, and it is a failure mode

The one harness effect that clears the noise is a loss, not a gain. OpenCode trails by up to 9 points on the large pool. On one model, roughly half of that gap comes from runs its output cap cut short: when the model's reasoning runs long, Claude Code prompts it to continue, while OpenCode ends the step.

The authors tried porting Claude Code's continuation prompt into OpenCode and could not reliably recover the gap, because the same overflow rarely triggers on a retry. The failure is real and it is specific, and it lives in the interaction between the harness and the model's output length, not in anything the leaderboard number says on its own.

This is the useful shape of a harness finding. It names a mechanism, it shows up in the traces, and it points at a fix. A bare pass-rate gap does none of that. When a harness does move the score, check first whether the gain comes from recovery, compaction, or tool reliability, the parts of the harness that stop a task from hard-failing. Those are the parts that earn their keep.

Finding 4: the bill is where the harness shows up

With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the fixed preamble of system prompt and tool schemas that each harness sends with every step: about 16,000 tokens in Claude Code, 7,000 in OpenCode, and under 1,000 in mini-SWE-agent. Per-call tool output and per-step growth differ far less. In practice, the bill is preamble size multiplied by the number of steps.

Harness	Preamble tokens sent with every step
Claude Code	about 16,000
OpenCode	about 7,000
mini-SWE-agent	under 1,000

The provider's price for cached input scales the bill and does not reorder it. So a harness can triple the cost per task while the score stays inside the noise band. That is the trade teams are actually making when they pick a harness on pass rate alone: they are buying tokens, and the score they think they bought is not in the receipt.

For a team running agents at volume, this is the finding that matters most. Compare dollars or tokens per solved task, not just pass rate. Ask whether the extra spend buys a difference larger than your rerun noise, and whether it buys anything at all.

Finding 5: most harness claims are below the resolution

At the observed flip rates, 45 tasks catch a 13-point gap only about half the time. You need roughly 447 tasks to resolve a 5-point difference. A vendor's claimed 2.4-point harness lead sits below what either sample size can separate.

This is the part of the paper that should change how teams read announcements. A harness release that claims a small gain on a small benchmark has not shown a gain. It has shown a number that the benchmark cannot resolve. The honest version of that announcement would include the rerun noise, and almost none of them do.

What this means for data and AI practitioners

1. Measure your own noise floor first. Run the same configuration twice on your own task mix. If the two runs disagree on a meaningful share of tasks, you do not yet have a baseline. One run is an anecdote.
2. Treat sub-five-point harness gaps as noise. On 447 tasks, five points is about the resolution limit. On 45 tasks, even a 13-point gap is a coin flip. Ask for the rerun data before you believe the gain.
3. Check failure modes before you conclude anything about the wrapper. Output caps, context limits, step limits, offline tool failures, and retry behavior can all produce a "harness difference" that is really a failure-mode difference.
4. Treat cost as a first-class metric. Compare dollars per solved task across harnesses, not just pass rate. A 16,000-token preamble, sent with every step, is a line item.
5. Fix the harness before you change the model. If you compare models under different harnesses, you cannot attribute the result to either one. Hold the wrapper constant, then vary the model. Or hold the model constant, then vary the wrapper.

In favour

The control is the whole paper, and it is done right. Ten rerun pairs on identical configurations give a noise floor that most benchmark papers never measure, and every claim in the paper is read against it. The cost analysis is equally careful: one price list, same model, same tasks, so the 3x gap is a harness effect and nothing else. And the OpenCode finding shows the method can detect a real harness effect when one exists, which makes the null results on the other two harnesses harder to dismiss.

Against

The scope is narrow. SWE-bench Verified is one benchmark family, and coding agents are only one kind of agent. The 45-task subset cannot resolve small gaps, and it is the subset that ran all five models. The closed anchor model, Claude Opus 5, ran only inside Claude Code, so the ceiling reference is confounded with the harness it ran in. And a preprint is a preprint: no peer review has checked the statistics yet.

What would make me wrong

A replication on a second benchmark family, with rerun pairs, that finds a consistent harness gap above five points would weaken the headline. So would a study showing the 13% flip rate is specific to SWE-bench Verified task design rather than agent runs in general. On the cost side, a harness that charges a large preamble but cuts steps enough to come out cheaper per solved task would complicate the bill argument. Until then, the burden is on the next harness claim to bring its rerun data with it.

Sources

• Liu, Y. and Han, Z. (2026). What Does a Harness Buy? Tokens, Mostly. arXiv:2610.04433.
• Devinios (2026). Switching agent harnesses may be noise: a 447-task study. DEV Community.

Related on Everyday Data Science:

• A True Tool Result Can Still Be Wrong: ReplayBench Found a 64.9% Error Rate
• Passed Once Is Not Reliable: ThinkingBox Grades Agents on the Database, Not the Demo
• The Hermes Index Ranks Agent Models on Score and Cost. The Lab That Built It Just Raised $90M.

When did you last change a harness, or a model, on the strength of a two-point leaderboard gain? And did you rerun it first?