Your Agent Tests Pass. They Mostly Test the Plumbing.

Benchmarks get the attention. The tests developers write for their own agent applications get almost none, and that is the gap this paper fills.

Tangent, from Rangeet Pan and colleagues at IBM T. J. Watson Research Center, Georgia Tech and the University of Georgia, asks a plain question: when teams build LLM-based agents, what do their tests actually check? The team mined GitHub for Python projects built on agent frameworks, found 1,190 candidate repositories using 11 frameworks, and extracted 1,723 agents and 4,111 tools. To keep the manual work honest, they narrowed the deep analysis to projects with substantial test suites, then hand-labelled 2,572 test methods from 240 modules across 12 projects. Two labellers worked independently, with a Cohen's kappa of 0.92. They also interviewed 10 senior industry practitioners, who averaged over 17 years of experience.

That method matters. This is counted evidence about real code, not a survey of intentions and not a vendor benchmark.

Finding 1: Most tests never run the whole agent

61.8% of the labelled tests evaluate a tool or an agent in isolation. 34.6% target interactions between agents or between an agent and other components. Only 3.5% cover multi-tool invocation, and 0.1% test the application through its exposed API.

Read that against how agents fail in production. Failures usually live in the interactions: the wrong tool chosen after a plausible first step, state carried across turns, a second tool receiving the output of the first in a shape nobody expected. The layer where those failures live is the layer with the thinnest test coverage.

The authors also found that the framework shapes the testing. Projects built on MCP and FastMCP, which give tools clear isolation boundaries, lean further toward unit tests. LangChain and LangGraph projects, where coordination is more explicit in the code, do more integration testing. The tooling you pick quietly decides what you will bother to test.

Finding 2: A third of tests mock the thing they claim to test

34.4% of tests mock an agent or a tool in some form: 21.7% mock selected components and 22.3% mock the entire agent (the categories overlap). Only 13.4% of mocking instances use custom test doubles built for a specific objective. The rest rely on standard patching and fixtures.

Mocking has a legitimate place. It makes tests fast and deterministic. The problem the paper names is specific to agents: a tool has two interfaces, a machine-facing API and a natural language description that shapes how the model reasons about it. A mock that returns the right value still says nothing about whether the description leads the model to call the tool at the right time, with the right arguments, or at all.

Finding 3: The inputs are easier than the job

39.9% of tests rely on simple, example-like inputs. Only 10.5% cover multi-step scenarios.

That is the test-data version of a familiar problem. A single clean prompt with an obvious answer will pass under almost any model version, prompt edit or tool change. It will also pass the week before a messy real request breaks the agent. If your test inputs are tidier than your users, your suite measures the tidy world.

Finding 4: Reasoning is the product, and it is barely asserted on

Most assertions target output (55.9%) and state (51.1%). Only 8.7% address reasoning-related behaviour, and the paper is blunt about what those few do: they usually check that reasoning-like behaviour is present, not that it was correct or useful. One example they cite verifies that subagent activity appears in the logs. It does not check whether delegating was appropriate or whether the subagent's work improved the result.

Only 4.1% of tests validate interactions with non-deterministic non-agentic components, and 2.4% with deterministic ones. For systems whose whole risk profile is side effects (files written, records updated, messages sent), that is a thin record on the part that can do damage.

Finding 5: Security, performance and cost are nearly untested

Only 7.5% of tests target non-functional requirements at all. Within that small slice, 49.0% focus on security, 33.5% on performance, 8.8% on memory and 4.6% on policy concerns.

So roughly 2.5% of all labelled tests touch performance, for a class of software where a single task can loop through dozens of paid model calls. The 10 practitioners interviewed placed more weight on non-functional testing than the open source projects did, which suggests production pain teaches what repositories do not show. Even so, the authors report the same gaps on both sides: no formal testing foundations, unclear test objectives, and difficulty generating good test data.

What this means for data and AI practitioners

You do not need to rebuild your suite. You need to rebalance it, and the study gives you the proportions to aim away from.

1. Add one end-to-end task test per workflow that matters. Real tools, a realistic messy input, and an assertion on the final state of the world, not just the final text. If the agent updates a record, check the record.
2. Assert on the trajectory, not only the answer. Which tools were called, in what order, with what arguments. A correct answer reached by calling the wrong tool with a customer's wrong ID is a failure your output assertion will miss.
3. Keep the mocks, and label what they cannot prove. A mocked-tool test proves orchestration logic. It proves nothing about tool selection driven by the tool description. Say so in the test name or docstring, so nobody reads green as end-to-end confidence.
4. Use inputs with dirt in them. Ambiguous requests, missing fields, a prior turn that changes the meaning, a tool that returns an error first. The study's 10.5% multi-step figure is the number to beat in your own suite.
5. Add one non-functional test. A token or cost ceiling per task, a timeout, or a check that a prompt-injection string in retrieved content does not trigger a tool call. One is infinitely more than the 92.5% of tests that cover none of this.

Against

The sample skews toward projects disciplined enough to have large test suites, so general practice could be worse than these numbers, or different in kind. The study counts encoded tests in repositories. It does not count observability, tracing reviews, eval dashboards or manual QA, which the authors acknowledge are real validation routes, even if they sit outside the test suite. And a unit-heavy split is not automatically wrong: cheap, fast unit tests should outnumber slow end-to-end tests in any healthy suite. The argument is about the near-absence of the other layers, not about the existence of the first one.

In favour

The labelling was manual, doubled and checked, with a kappa of 0.92, which is near-perfect agreement. The taxonomy is concrete: 23 testing patterns across fixtures, data, objectives and assertions, so the percentages come from defined categories rather than vibes. The practitioner interviews point the same way as the repository data on the core gaps, which is the kind of triangulation most empirical software studies do not get. And the paper was presented at ASE 2026, a peer-reviewed venue, rather than posted as a blog claim.

What would make me wrong

• If a comparable hand-labelled study of production, closed-source agent systems found reasoning assertions and non-functional tests at several times these rates, the open source picture would stop being a useful warning.
• If teams with unit-heavy, mock-heavy suites showed production incident rates no worse than teams with trajectory and state-based testing, the missing layers would matter less than this reading suggests.
• If framework-level tooling ships trajectory assertions and state diffing as defaults, and adoption data shows developers actually use them, this gap becomes a 2026 snapshot rather than a standing problem.

Sources

• Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications (arXiv:2608.08413), Pan et al., ASE 2026. All percentages, counts and the dataset description above are from the paper's abstract, dataset section and findings.
• Tangent HTML full text (arXiv), testing-level distribution (Section 3), mocking and assertion patterns (Section 4), test complexity medians (Section 5), practitioner interviews (Section 6).

Related on Everyday Data Science:

• ThinkingBox: agents graded on database state
• A 100-line agent eval harness that catches regressions
• MCP, explained for data people

If you opened your own agent test suite today, what share of the tests would still pass if the model stopped reasoning well but the plumbing kept working?