ToolGrad Starts With the Answer, Then Writes the Question
The 99.8% result is a clever data-generation win—not proof that small models can run production agents.
Google Research highlighted a deceptively simple idea on September 10: if you want training data for tool-using models, stop beginning with the user’s question. Build a valid tool chain first. Then write the question that chain can answer.
That inversion is the core of ToolGrad, an ACL 2026 Findings paper whose current arXiv version was revised in June. The timing matters: this is a newly published research explanation, not a brand-new experiment conducted yesterday.
The reported result is striking. ToolGrad produced valid synthetic examples at a 99.8% pass rate, versus 63.8% for a query-first depth-first-search baseline, while using fewer tool-execution steps and creating longer ground-truth tool chains. A Gemma 3 12B model fine-tuned on only 500 ToolGrad examples nearly matched Gemini 2.5 Pro on the paper’s Berkeley Function Calling Leaderboard evaluation.
But the interesting story is narrower than “small agents beat frontier models.” This is a lesson about how we manufacture supervision. It is also a reminder that a benchmark win can be both real and easy to overgeneralize.
The usual pipeline starts with an unsolved problem
Most synthetic tool-use datasets follow a query-first pattern:
- Sample a set of APIs.
- Ask a model to invent a user request involving those APIs.
- Search for a sequence of tool calls that solves the request.
- Keep the successful trace as training data.
The weakness is visible in step three. The generated question may be ambiguous, impossible, or unsupported by the available tools. A search agent can explore many dead ends before discovering that no valid path exists. Worse, a successful trace may still include irrelevant or failed calls that become part of the supposed “ground truth.”
This is not merely a compute problem. It is a label-quality problem. If the trajectory is noisy, supervised fine-tuning teaches the model to imitate that noise.
ToolGrad reverses the direction
ToolGrad constructs the executable workflow first:
API pool → propose candidates → execute them → select a useful success
→ extend the workflow → rewrite the query and response
Each iteration has four parts.
The API proposer inspects a mini-batch of 50 tool definitions and suggests up to three candidates that could extend the current workflow.
Parallel executors actually call those candidates and return execution reports. This filters proposals through the tools rather than through the model’s confidence alone.
The selector chooses the most useful successful call and decides where it belongs in the chain. The authors describe this selection as a textual “gradient”: not a mathematical derivative, but directional feedback about how to improve the example.
The updater adds the chosen API and rewrites the synthetic user query and final response so they remain consistent with the now-expanded tool chain.
The loop runs for ten iterations in the reported setup. Afterward, similar but incorrect tools are added as negatives, making the selection problem more realistic than presenting only the correct tool.
The philosophical move is simple: generate from what is known to work instead of asking an agent to discover a solution to a question that may never have been solvable.
What the numbers actually show
Against the paper’s query-first DFS baseline, ToolGrad reported:
- 99.8% vs. 63.8% pass rate for generated examples
- 3.4 vs. 2.1 ground-truth tool uses per example
- 20.0 vs. 34.3 tool-execution steps during generation
- nearly unchanged LLM invocation counts: 63.9 vs. 64.5
That last line is important. The cost improvement is not magic. ToolGrad does not eliminate model calls; it spends its work more productively by reducing failed tool exploration.
The team then generated ToolGrad-500 with Gemini 2.5 Flash-Lite and used the 500 examples to fine-tune Gemma 3 models at 1B, 4B, and 12B parameters. On BFCL’s single-turn tool-use tasks, the paper reports gains of 8.1, 8.0, and 6.3 points over the corresponding base models. Google’s research post reports an overall score of 83.1 for the 12B model, 0.1 behind Gemini 2.5 Pro under the study’s comparison.
The unseen-tool result matters. The training APIs came from ToolBench, while the BFCL evaluation had minimal tool-definition overlap. That makes the improvement more interesting than simple memorization of API names.
Still, “unseen tools” is not the same as “unseen production conditions.”
Why this is a data-engineering result disguised as an agent result
The strongest contribution is not a new agent loop at inference time. It is a better factory for producing supervised examples.
Tool use requires three kinds of competence:
- deciding whether a tool is needed
- selecting the right function and arguments
- recovering across several steps when the world responds unexpectedly
ToolGrad provides strong supervision for the first two inside a single-turn format. It creates examples where the chosen chain has already been executed and verified. That can improve signal quality dramatically, even with a small dataset.
This resembles a good data-engineering principle: validate before promotion. Instead of storing a synthetic trajectory because an LLM narrated it convincingly, ToolGrad promotes calls that produced successful execution reports.
Where the headline breaks
Three limitations change how we should interpret the result.
1. This is not a full agent benchmark
The evaluation focuses on BFCL v1/v2 single-turn tool use. The paper states that multi-turn tool use and the broader agent tasks in BFCL v3/v4 are out of scope. ToolGrad’s fine-tuned models predict their tool uses in one shot; they are not being tested on the same iterative reason-act-observe loop used by many production agents.
A model can format a correct function call and still fail when credentials expire, a user changes their mind, a tool returns partial data, or a later step depends on an earlier observation.
2. Synthetic questions may not sound like humans
Because the workflow is known before the query is written, the generated question is guaranteed to fit the tools. That is the source of the high pass rate—and a possible source of artificiality. The authors acknowledge that generated queries may not match real human intentions or linguistic diversity.
Real requests are underspecified. People omit units, mix goals, refer to earlier context, and ask for actions the available tools cannot complete. An answer-first dataset can underrepresent exactly those hard cases.
3. More synthetic data did not keep helping
In the scaling study, the 4B model improved and then declined as the dataset grew from 100 toward 2,000 samples. The authors attribute the plateau partly to repetitive tool-use patterns and the lack of shared memory across independent generation runs.
This is a valuable negative result. “Synthetic data scales cheaply” does not imply “synthetic diversity scales automatically.” If the generator keeps rediscovering the same workflow shapes, a larger dataset can become a louder version of the same lesson.
What practitioners should copy
You do not need to reproduce the entire framework to borrow its best ideas.
Start from verified capabilities. Inventory the operations your tools can actually complete, including required parameters and failure modes.
Execute before labeling. Do not mark a tool trajectory as correct because a model says it is correct. Run it in a sandbox or deterministic test environment and preserve the trace.
Generate negatives deliberately. A model that always sees the right tool beside unrelated distractors does not learn fine-grained selection. Include plausible near-misses.
Separate syntax from recovery. Evaluate correct function names and arguments, but also test multi-turn recovery, clarification, permission boundaries, and partial failures.
Mix synthetic and human demand. Answer-first generation supplies clean, solvable examples. Real support tickets, analyst requests, and user studies reveal ambiguity and distribution shift. You need both.
Track workflow diversity. Count unique tool combinations, chain shapes, domains, and argument patterns. A dataset with 10,000 rows can still contain only a few distinct lessons.
A better evaluation ladder
Before calling a fine-tuned model “production-ready,” test it at four levels:
- Call correctness: Did it select the correct tool and produce valid arguments?
- Chain correctness: Did several calls occur in the right dependency order?
- Interaction robustness: Can it clarify ambiguity and recover from tool errors across turns?
- Outcome reliability: Did the real task finish without violating business or safety constraints?
ToolGrad offers compelling evidence at the first two levels under the reported conditions. It does not yet settle the last two.
What would make me wrong
My cautious reading would be too conservative if three things happen.
First, answer-first datasets could retain their advantage on BFCL v3/v4, tau-bench-style environments, and long-horizon tasks with stateful failures. That would show the method teaches more than clean single-turn function selection.
Second, adding memory or diversity objectives could remove the small-data plateau without sacrificing the 99.8% validity rate. Then the framework would become a credible route to large-scale agent curricula rather than a sharp technique for compact supervised sets.
Third, blinded human evaluation on naturally collected requests could show that answer-first prompts match real intent as well as human-written data. That would weaken the concern that solvability is being purchased with synthetic neatness.
Until then, the correct conclusion is modest but useful: ToolGrad demonstrates that changing the order of data generation can improve both validity and downstream tool-call performance. It does not demonstrate that a 12B model trained on 500 examples can replace a production agent system.
The bigger lesson
Agent capability is often treated as a model problem. ToolGrad is evidence that it is also a dataset-design problem.
When training examples begin with questions that may not be solvable, the pipeline spends money searching, failing, and sometimes labeling noisy traces as truth. When examples begin with verified actions, the supervision becomes cleaner.
The answer-first idea is clever because it turns tool execution from the end of annotation into the foundation of annotation. The remaining challenge is to preserve the messiness of real users after you make the labels clean.
Sources
Found this useful? Passing it on to someone who builds is the best way to help the publication grow.