State of Coding Agents: Who Actually Wins on Real-World Tasks?
What the SWE-bench leaderboard measures, and what a randomized trial found instead
Coding agents now score above 90% on SWE-bench Verified. The leaderboard says the problem is essentially solved.
In a randomized controlled trial, experienced developers given AI tools completed real tasks 19% slower: and estimated afterwards that they'd been 20% faster.
Both of those are true. Understanding why is the most useful thing you can know about coding agents right now.
What the leaderboard says
SWE-bench Verified, real GitHub issues from real repositories, with real tests, was supposed to be the benchmark that couldn't be gamed. Top agents now clear 90% on it, and the frontier models trade places at the top every few weeks.
Taken at face value, that means an agent resolves nine out of ten real software issues unaided.
Nobody who has actually shipped with one believes that. So where does the number come from?
Then you read the audits
This is where it gets uncomfortable, and where most "state of coding agents" write-ups stop.
| What auditors found | Figure |
|---|---|
| Successful patches involving solution leakage | 32.67% |
| Models recalling correct file paths from training data | up to 76% |
| "Solved" tasks that are semantically incorrect: passing by coincidence or reward-hacking the harness | 19.78% |
| Tasks in SWE-bench Pro found to be broken | ~30% |
Read the third row again. On top-30 leaderboard entries, roughly one in five "solutions" doesn't actually solve the problem. It satisfies the test without producing correct code.
And the first two rows explain the rest. These are public GitHub issues. The fixes are on the internet. The models were trained on the internet. When an agent "finds" the right file, it is often not reasoning about your architecture, it is remembering.
OpenAI has been blunt enough to say the eval no longer provides meaningful signal on software-development capability.
The measurement that should change your mind
METR did what almost nobody does: a randomized controlled trial. Sixteen experienced open-source developers, 246 real tasks, in mature repositories they already knew well. For each task, AI assistance was randomly allowed or forbidden.
The developers expected a speed-up. So did METR.
They were 19% slower with AI. And they believed they'd been 20% faster.
That 39-point gap between perception and measurement is the single most important number in this field, because it means you cannot trust your own sense of whether these tools are helping you. The feeling of velocity and the fact of velocity came apart completely.
Why the gap exists
Once you hold both results together, the explanation is almost mundane.
The benchmark hands the agent everything hard about the job already solved. The problem is specified. The repo is known. The test that defines "correct" exists. Someone already wrote the right answer, and it's in the training data.
Real work is the opposite. The ticket is ambiguous. The codebase is private and inconsistent. There's no oracle. And critically, you have to read what the agent wrote.
That last one is the whole story. An agent produces a plausible 200-line diff in forty seconds. Verifying that it's correct, actually reading it, in a codebase you're responsible for, can take longer than writing eighty correct lines yourself would have. Reviewing confident, plausible, subtly-wrong code is one of the slowest activities in software. The benchmark never charges the agent for it. Your afternoon does.
This is why the leaderboard ranking inverts on real tasks. Cognition's Devin 2.0 sits at 45.8% on SWE-bench, roughly half the top score, yet reportedly beats leaderboard winners on the things engineers actually care about: long-horizon multi-file refactors, debugging genuinely novel bugs, navigating a codebase nobody has seen.
So who actually wins?
The honest answer is that the question is malformed. There is no single winner, because "real-world tasks" isn't one thing. What predicts the outcome is the shape of the task, not the ranking of the agent.
| Task shape | Agents are… |
|---|---|
| Well-specified, greenfield, boilerplate, scaffolding | Genuinely transformative |
| Unfamiliar codebase, exploratory, "where does this even live" | Strong: search and orientation is a real superpower |
| Test writing, migrations, mechanical refactors | Strong |
| Novel bug in a mature codebase you know well | Often a net loss: this is exactly METR's condition |
| Ambiguous requirements where the hard part is deciding what to build | Not the bottleneck |
Notice that the row where agents lose is the row where you are already an expert. That's not a coincidence, it's the mechanism. The more context you carry in your head, the less an agent adds, and the more you pay in verification.
How to actually evaluate one, for your codebase
Stop reading leaderboards. Build a five-task benchmark of your own, it takes an afternoon and it will tell you more than every leaderboard combined.
- Pull five recently-merged PRs from your own repo. Real ones, already reviewed and shipped.
- Hand the agent the original issue text. Nothing else, no hints, no pointing at files.
- Measure time-to-merged, not time-to-first-draft. The draft is free. The review is the cost.
- Count the review burden: how many rounds did it take, and how many bugs did you catch?
- Compare against the actual human time logged on that PR.
Your five PRs are not in anyone's training data. That's precisely what makes them worth more than SWE-bench.
Key takeaways
- Benchmark scores are contaminated. ~33% solution leakage, up to 76% file-path recall, and ~20% of "solves" are semantically wrong.
- A 90%+ SWE-bench score does not mean 90% of your tickets. The benchmark pre-solves the hard parts of the job.
- METR's RCT is the number to remember: 19% slower, 20% felt faster. You cannot self-assess this.
- The hidden cost is verification. Reading plausible-but-wrong code is slower than writing correct code.
- Agents win where you're weak and lose where you're strong. Boilerplate and unfamiliar code: huge. Novel bugs in code you know cold: often a net loss.
- Benchmark on your own merged PRs. They're the only test set that isn't in the training data.
Have you actually measured whether agents make you faster, or are you going on how it feels? If you've timed it, I'd really like to hear what you found. 👇
Sources: METR, arXiv.09089 (RCT; METR now labels this result historical). SWE-bench contamination and correctness figures are from independent audits of the Verified leaderboard, incl. SWE-Bench+ (arXiv.06992) and subsequent leaderboard analyses; OpenAI has separately published on separating signal from noise in coding evaluations. Leaderboard scores move constantly, treat any specific figure as a snapshot.
The Agent Revolution Is Here, and Most Organizations Are Not Ready
Next →RAG vs. Fine-Tuning: A 2026 Decision Framework for Practitioners
Found this useful? Passing it on to someone who builds is the best way to help the publication grow.