Everyday Data Science
Latest
Agentic workflows now power a third of surveyed enterprise automationAfrica's AI startup ecosystem posts record funding yearNew benchmark results reshape the coding-agent leaderboardNigeria launches national AI strategy with major investment planRwanda's sovereign AI cloud enters public betaThe future of AI agents: from tools to teammates
Agentic AIBenchmark Watch

Multi-Agent Systems Can Now Write Their Own Workflows. The Best One Still Fails 63% of Long Tasks

WorkflowArena hands 145 long-horizon tasks to nine models and lets each one organize its own workers. Claude Opus 5.5 at maximum effort resolves 36.95%. The more interesting result is that a good workflow, handed to a weaker model, is worth 6.4 points on its own.

Ibrahim Denis FofanahIbrahim Denis FofanahData Scientist & AI Researcher15 min read·Building Agentic AI Systems · Benchmark Watch
Part 8 of 11Building Agentic AI SystemsView path →

Most agent benchmarks treat the agent as one unit. They hand it a task, watch it work, and score the result. That made sense when an agent was a single model in a single context. It no longer describes how the strongest systems actually work. The current generation of harnesses lets the model spawn worker agents and write the executable workflow that coordinates them. The structure of the system, once fixed by its developers, is now decided by the model at run time.

WorkflowArena, submitted to arXiv on November 9, 2026 by researchers at Navers Lab, Einsia.AI, and Tsinghua University (arXiv:2611.12856), measures that decision directly. It gives agents 145 long-horizon tasks across 8 professional domains and 17 task forms, from software engineering and formal mathematics to legal analysis and 3D production. Each task starts from a versioned professional artifact or an authored scenario with fully specified dependencies, and asks for an integrated deliverable made of interdependent components: clause-level analyses that must be reconciled across a contract portfolio, or proofs that build on earlier results. A hidden verifier scores the submission against the task contract, including checks that individually correct but unreconciled components cannot pass. The agent decides whether to delegate at all, and the time it spends organizing the work comes out of the same budget as the work itself. Every delegation passes through a recorded interface, so each outcome can be read together with the workflow that produced it.

A task counts as resolved only when its submission satisfies every acceptance criterion of its domain. The headline number, Overall Resolved, is the equal-weight mean of the 8 domain resolved rates.

Finding 1: the ceiling is 36.95%, and the domain spread is the story

Twenty-five configurations of nine models ran all 145 tasks. The strongest, Claude Opus 5.5 in Claude Code at maximum effort, resolves 36.95% of tasks on average across domains. The second best, Opus at high effort, resolves 34.11%. No configuration leads every domain.

Configuration Overall SWE OPS SCI MATH FIN LAW CDN 3D
Claude Opus 5.5, Claude Code, max 36.95 29.41 22.22 56.25 95.83 12.50 0.00 57.14 22.22
Claude Opus 5.5, Claude Code, high 34.11 17.65 16.67 50.00 87.50 18.75 4.55 50.00 27.78
Claude Sonnet 5.5, Claude Code, max 33.21 11.76 11.11 50.00 79.17 12.50 9.09 64.29 27.78
GPT-6 Astra, Codex, xhigh 30.80 0.00 22.22 50.00 83.33 12.50 4.55 57.14 16.67
GPT-6.1 Sol, Codex, xhigh 29.50 0.00 16.67 43.75 79.17 12.50 4.55 57.14 22.22

The domain columns tell you where the ceiling actually sits. Formal mathematics is close to saturation: the best configuration resolves 95.83% of MATH tasks. Software engineering tops out at 29.41%. Law and finance are the hard floor: no configuration resolves more than 9.09% of LAW tasks or more than 18.75% of FIN tasks. Opus at maximum effort resolves zero LAW tasks while leading SWE, SCI, and MATH. Sonnet at maximum effort leads LAW and CDN and shares the lead in 3D.

The bottom of the table is just as informative. Kimi K3.1 Pro at maximum effort resolves 16.48% overall. GPT-6 Luna resolves 13.32%. DeepSeek V4.1 Flash resolves 6.08%, and GLM-5.3 Flash 4.43%. Doubao-Seed-Evolving resolves 0% of MATH tasks, as do DeepSeek and GLM. The gap between the strongest and weakest frontier-adjacent models on this benchmark is not a few points. It is a factor of eight.

The honest reading is that long-horizon, multi-component work is still mostly unsolved. A system that fails nearly two of three tasks is not a system you hand a contract portfolio to unsupervised. The interesting question is what separates the configurations that do better from the ones that do worse, and WorkflowArena records enough to answer it.

Finding 2: more effort pays for Claude, barely for GPT

Each of the four main models ran at five effort settings: low, medium, high, extra-high, and maximum. Both Claude models score best at maximum effort, and both GPT models score best at extra-high. But the size of the effort effect differs sharply between the two families.

Opus ranges from 22.02% at low effort to 36.95% at maximum, a spread of nearly 15 points. Sonnet ranges from 16.88% to 33.21%, a spread of more than 16 points. Astra varies only between 27.34% and 30.80% across all five settings, and Sol between 25.63% and 29.50%. More effort is not even uniformly beneficial for the GPT models: Sol at low effort resolves more tasks than at medium or high effort.

Token spend shows the same split. Opus at high effort resolves 34.11% of tasks using 7.36 million tokens per task, a rate no other model reaches at any token budget. Sonnet at maximum effort uses 3.2 times the tokens of its high-effort configuration and gains 7.95 points. Sol at extra-high effort uses 1.9 times the tokens of its low-effort configuration and gains only 1.91 points.

If you are choosing an effort setting, the practical rule from this table is simple. For the Claude systems, effort is a real lever: low effort Opus scores below low effort Sol, and maximum effort Opus leads the whole table. For the GPT systems in Codex, effort is mostly a cost dial. Turning it up spends more tokens for about the same score, and sometimes for a lower one.

Finding 3: Claude delegates rarely in large groups, GPT delegates often in small units

WorkflowArena records every delegation, so the paper can describe how each system organizes work. At high effort, the two families look like opposites.

System Workflow calls per task Worker invocations per Workflow
Claude Opus 5.5, Claude Code 0.86 5.55
Claude Sonnet 5.5, Claude Code 0.76 6.14
GPT-6 Astra, Codex 2.09 1.72
GPT-6.1 Sol, Codex 1.94 1.69

Opus and Sonnet launch fewer than one workflow per task on average, but each workflow carries five or six workers. Astra and Sol launch about two workflows per task, with fewer than two workers each. Among tasks with at least one workflow call, a single call accounts for 70% of Opus executions and 68% of Sonnet executions, against about 32% for Astra and Sol. Groups of nine or more workers make up 24% of Opus workflows and 30% of Sonnet workflows, while about 99% of Astra and Sol workflows contain at most four workers.

Effort moves the two families differently too. From low to maximum effort, Astra's calls per task rise by 67%, from 1.33 to 2.22, while its workers per workflow change by only 5%. Sol's calls rise by 79%, from 1.12 to 2.00, with a 12% change in worker count. The Claude systems instead reach their highest call frequency at extra-high effort: Sonnet moves from 2.33 calls and 4.15 workers per workflow at extra-high to 1.57 calls and about 4.35 workers at maximum. At the top end, Claude Code systems consolidate. Codex systems multiply.

The domain shapes delegation inside each system as well. In LAW, Opus and Sonnet average 14.29 and 8.45 worker invocations per workflow, against 1.73 for Astra and 1.66 for Sol, while their call frequencies show the reverse ordering. In MATH, Kimi and Luna make 9.67 and 6.46 calls per task but average only 1.74 and 1.26 workers per workflow, although Kimi's groups grow to 5.84 and 5.08 workers in LAW and FIN. Delegations in OPS stay small for every model, at 0.17 to 1.72 calls per task and 1.00 to 2.00 workers per workflow. Call frequency alone does not characterize the scale of collaboration. A system can delegate constantly and still be working in pairs.

The 3D case study makes the difference concrete. Three Blender scene tasks, a palace, a cathedral, and a fortress, each requiring renders and a manifest reconciling construction drawings with a change order, show the same split in miniature. In the palace, Opus runs seven builders followed by assembly and iterative repairs, while Astra first establishes a shared foundation and then assigns assembly to a single worker. In the cathedral, Opus makes two sequential whole-scene attempts, while Astra divides the scene among three regional builders and a separate assembly worker, which introduces interfaces at shared structural positions. In the fortress, both rely on a single whole-scene builder, but Opus precedes construction with seven drawing-analysis calls, while Astra audits the drawings concurrently with construction. Delegation serves two distinct purposes across these tasks: building components in parallel, and preparing the inputs of a central builder.

Finding 4: price reorders the ranking

Token counts and dollars tell different stories, because per-token prices differ by more than an order of magnitude across these models. The token frontier consists entirely of Opus configurations. The cost frontier is far more mixed, running from Luna through low-effort Sonnet and several Sol configurations to Opus.

The numbers that matter for a buyer: Sol at low effort resolves 27.59% of tasks at $3.61 per task. Opus at high effort resolves 34.11% at $6.66. Astra at extra-high effort resolves 30.80% at $29.26, which is 3.31 points below Opus at high effort at 4.4 times its cost.

That is the kind of result that should change a purchasing decision. The most expensive configuration in the table is not the best, and it is not close to the best. It costs more than four times the price of the leader for a worse score. If you are running long-horizon agent work on a budget, the leaderboard order and the cost order are different lists, and this paper gives you both.

Finding 5: a good workflow is worth 6.4 points, even to a model that did not write it

The headline table mixes two things: how good a model is at the work, and how good it is at organizing the work. WorkflowArena separates them with a transfer experiment. On a challenging subset of 32 tasks, the executor is held fixed and only the source of the workflow varies. Astra and Opus write workflows at maximum effort, and Kimi, DeepSeek, Luna, and Doubao execute them. The source model never edits the workspace, so every gain comes from the workflow it authors, including the instructions it gives each worker.

Execution model Own workflow Opus workflow, static Opus workflow, dynamic
Kimi K3.1 Pro 35.75 45.04 46.61
GPT-6 Luna 25.08 30.37 25.35
Doubao-Seed-Evolving 20.39 28.04 27.46
DeepSeek V4.1 Flash 18.37 21.79 25.24

Averaged over the four executors, Opus workflows add 6.41 points in the static mode, where the workflow is written once and frozen, and 6.27 points in the dynamic mode, where the source revises the workflow as workers report back. Kimi improves from 35.75 to 46.61 with Opus dynamic workflows. DeepSeek improves from 18.37 to 25.24. Luna and Doubao get their highest scores with Opus static workflows, at 30.37 and 28.04.

Workflows from Astra transfer far less well, averaging minus 2.53 points in the static mode and plus 1.34 in the dynamic mode. They lower the scores of Kimi and DeepSeek in both modes. Yet Astra is itself a far stronger system than any of the four executors, resolving 27.34% of tasks at maximum effort against at most 16.48% for the executors. Strong solo performance does not by itself yield workflows that help other models. The ability to orchestrate others is not simply a byproduct of the ability to complete the task.

That is the result with the most practical weight in the paper. If orchestration transfers across models, then a strong orchestrator paired with cheap executors is a real architecture, not a demo trick. A 6.4-point lift from the workflow alone is larger than the gap between several neighboring configurations in the main table. It is also a result that cuts against the usual way teams pick models: the best executor is not necessarily the best author of the plan, and the best author may be worth paying for even when a cheaper model does the work.

What this means for data and AI practitioners

  1. Score the organization, not just the output. If your agents delegate, log the workflow the way WorkflowArena does: calls per task, workers per workflow, and the outcome together. A system that scores well with a bad workflow is hiding a failure mode you will meet on a harder task.
  2. Match the effort setting to the model family. For the Claude systems here, effort is a real lever worth paying for. For the GPT systems in Codex, extra effort mostly buys tokens. Check your own numbers before assuming a higher setting helps.
  3. Price the configuration, not the model. The most expensive run in this table costs 4.4 times the leader and scores lower. A $3.61-per-task configuration resolves 27.59%. Build your own cost frontier before you commit to a vendor default.
  4. Try a strong orchestrator with cheap executors. The transfer experiment says the plan is worth about 6.4 points to a weaker model. If you run multi-agent work at volume, test whether a frontier model writing the workflow and a cheap model executing it beats a mid-tier model doing both.
  5. Do not trust a multi-agent demo on math tasks. MATH is the one domain approaching saturation at 95.83%. LAW tops out at 9.09% and FIN at 18.75%. Evaluate on the domain you actually ship in.
  6. Expect law and finance to stay hard. The two domains with the strictest reconciliation requirements are the two where every model fails. If your use case is contracts or financial analysis, budget for human review and verification steps in the workflow itself, not as an afterthought.

In favour

The benchmark measures something new and does it honestly: organization is consequential, autonomous, integration-sensitive, cost-accounted, and observable, and the paper states those five requirements up front and shows how the design meets each one. The transfer experiment is the strongest part of the work, because it isolates the variable everyone else leaves confounded. The delegation analysis gives practitioners a vocabulary, calls per task and workers per workflow, that they can apply to their own traces tomorrow. The cost analysis is unusually complete for a benchmark paper, with per-task dollar figures that make the ranking actionable.

Against

One run per task with no variance estimates means the middle of the table is noise as much as signal: Opus at high effort (34.11%) and Sonnet at maximum (33.21%) are within a couple of points of each other with no way to say which is truly better. The model-versus-harness confound is the paper's own stated limitation, and it matters, because the headline Claude-versus-GPT delegation split could be a Claude Code versus Codex split. An LLM judge grades the content and 3D domains with no human agreement check, and those domains include the two where the paper claims the sharpest domain effects. The benchmark is also vendor-adjacent: it is funded and supported by Navers Lab and Einsia.AI, and the project homepage is theirs. That does not invalidate the numbers, but it is a reason to want an independent replication before the ranking hardens into received wisdom.

What would make me wrong

  • A replication with three runs per configuration that puts Opus at maximum effort below 30% or above 45% would show the ceiling is mostly noise, not signal.
  • If a second harness for the same models reproduces the Claude-versus-GPT delegation split, the split is a model property. If it disappears, it was a harness artifact.
  • If human graders disagree with the Gemini judge on the content and 3D domains, the domain spread, especially the MATH versus LAW gap, needs re-measuring.
  • If Opus workflows fail to lift executors on a larger transfer set, the 6.41-point gain was a 32-task artifact.
  • If a cheaper executor with an Opus workflow beats Opus doing the whole job itself on a real workload, the orchestration story is even stronger than this paper claims.

Sources

Related on Everyday Data Science:

Would you pay for the model that writes the plan, or the model that does the work, if they are no longer the same model?

About the writer

Ibrahim Denis Fofanah
Ibrahim Denis Fofanah

Data Scientist & AI Researcher

3 followers

Data scientist and AI researcher at Pace University. I coined Artificial Frictional Unemployment, and built the first machine learning model for crop yield prediction in Sierra Leone. Author of Understanding Agentic AI. I write about agentic systems and applied ML, with a bias toward what actually works, and who gets left out when it doesn't.

Share

Found this useful? Passing it on to someone who builds is the best way to help the publication grow.

Built something worth sharing? Write it up for us →