Stop Eyeballing Your Agent's Answers. A 100-Line Eval Harness Catches the Regressions You Miss.

I once fixed an agent's median calculation on a Friday afternoon, shipped it, and broke its error handling in the same commit. The median was correct. Nobody noticed the error path was gone until a user hit it on Monday. I had tested the fix by hand, watched the right number come out, and called it done. I tested the thing I changed. I never re-ran the things I did not change.

That is the whole argument for an evaluation harness. An agent is a program whose behavior you cannot fully predict, which means every change can move something you were not looking at. Without a fixed set of cases and a repeatable score, "it works" means "it worked on the three prompts I tried just now."

Yesterday's research brief made the academic version of this point: VRL-Bench found that published trial-and-error methods like Reflexion beat a plain retry on some models and lose to it on others, and the only honest way to know is a matched comparison. This tutorial is the practitioner version. You do not need a 16,946-trial benchmark. You need 100 lines of Python, a fixed case set, and the discipline to run it before every commit.

What the harness does, and what it does not

The harness answers one question: does this version of the agent still do everything the last version did, plus the new thing? It does that by running the same inputs through the agent and scoring the outputs with small, strict check functions.

What it does not do is judge quality the way a human would. It will not tell you the agent's tone improved or its reasoning got more elegant. It catches regressions, which are the failures that cost you users. Save the vibes for the demo call.

Three design choices keep it honest:

1. Checks return evidence, not just booleans. Every check returns (passed, detail) where detail is a string like "got 2.8, expected 3.0". When a case fails at 2 a.m., the detail tells you why without re-running anything.
2. The harness survives agent crashes. A case that raises an exception is recorded as a failure with the exception type, not as a harness crash. An agent that throws on empty input is telling you something; the harness should write it down.
3. Results are saved as JSON. eval_results.json is your audit trail. Diff it across versions and the regression shows up as a line change, the same way code diffs work.

The harness, in full

This is the entire thing. It takes any callable as the agent, which means it works whether your agent is a function, a LangChain chain, or an HTTP call to your own API.

import json, time
 
class EvalHarness:
    def __init__(self, agent):
        self.agent = agent
        self.cases = []
 
    def add(self, name, task, check):
        """check(output) -> (passed: bool, detail: str)"""
        self.cases.append((name, task, check))
 
    def run(self):
        results = []
        for name, task, check in self.cases:
            start = time.perf_counter()
            try:
                output = self.agent(task)
                passed, detail = check(output)
            except Exception as e:  # harness survives agent crashes
                passed, detail = False, f"raised {type(e).__name__}: {e}"
            results.append({
                "name": name, "passed": passed, "detail": detail,
                "seconds": round(time.perf_counter() - start, 3),
            })
        return results
 
    def report(self, results, path="eval_results.json"):
        passed = sum(r["passed"] for r in results)
        print(f"{passed}/{len(results)} passed")
        for r in results:
            mark = "PASS" if r["passed"] else "FAIL"
            print(f"[{mark}] {r['name']} ({r['seconds']}s) - {r['detail']}")
        with open(path, "w") as f:
            json.dump(results, f, indent=2)

Nothing here requires a framework. If you are already using one, wrap your agent's invoke in a plain function and pass it in. The harness does not care how the answer was produced. It cares that the answer is the same, or better, than last time.

Writing cases that actually test something

A case is a triple: a name, an input, and a check. The name is the behavior contract in plain words. The input is the smallest realistic example of that behavior. The check is strict about the property you care about and lenient about everything else.

Here is a case set for a small data-question agent, the kind of helper that sits inside a larger analytics workflow:

def build_cases(harness):
    harness.add(
        "median_of_five_values",
        {"kind": "median", "values": [3, 1, 4, 1, 5]},
        lambda out: (abs(out["answer"] - 3.0) < 1e-9,
                     f"got {out['answer']}, expected 3.0"),
    )
    harness.add(
        "top_category_by_sales",
        {"kind": "top_category",
         "sales": {"maize": 40, "rice": 90, "cassava": 55}},
        lambda out: (out["answer"] == "rice",
                     f"got {out['answer']!r}, expected 'rice'"),
    )
    harness.add(
        "deadline_extraction",
        {"kind": "deadline", "text": "proposal deadline: 2026-10-05"},
        lambda out: (out["answer"] == "2026-10-05",
                     f"got {out['answer']!r}"),
    )
    harness.add(
        "graceful_empty_input",
        {"kind": "empty_input"},
        lambda out: ("error" in out,
                     f"got {out!r}, expected an 'error' key"),
    )

Three rules make cases worth keeping:

1. One behavior per case. The median case tests the median. It does not also test formatting. When it fails, you know exactly which behavior broke.
2. Check the contract, not the implementation. The empty-input check asks for an "error" key, not a specific message string. You want the freedom to reword the message without failing the suite.
3. Include at least one adversarial case. The empty input is not a realistic user query. It is the input that arrives at 2 a.m. from a broken upstream system. If your agent throws instead of answering, you want to know in the harness, not in the incident channel.

Watch it catch a regression

Here are two versions of the mock agent. Version 1 has a genuine bug: it returns the mean (2.8) instead of the median (3.0). Version 2 fixes the median but, in the same edit, starts raising a ValueError on empty input instead of returning a clean error dict. This is exactly the shape of my Friday-afternoon incident: fix one thing, break another, test only the fix.

import statistics
 
def agent_v1(task):
    if task["kind"] == "median":
        return {"answer": statistics.mean(task["values"])}  # BUG: mean, not median
    if task["kind"] == "top_category":
        return {"answer": max(task["sales"], key=task["sales"].get)}
    if task["kind"] == "deadline":
        return {"answer": task["text"].split("deadline: ")[1].strip()}
    if task["kind"] == "empty_input":
        return {"error": "no input provided"}
    raise ValueError("unknown task kind")
 
def agent_v2(task):
    if task["kind"] == "median":
        return {"answer": statistics.median(task["values"])}  # fixed
    if task["kind"] == "top_category":
        return {"answer": max(task["sales"], key=task["sales"].get)}
    if task["kind"] == "deadline":
        return {"answer": task["text"].split("deadline: ")[1].strip()}
    if task["kind"] == "empty_input":
        raise ValueError("empty input not supported")  # REGRESSION
    raise ValueError("unknown task kind")

Running the harness against both versions produces this (verified output from the actual run):

== agent v1 ==
3/4 passed
[FAIL] median_of_five_values (0.0s) - got 2.8, expected 3.0
[PASS] top_category_by_sales (0.0s) - got 'rice', expected 'rice'
[PASS] deadline_extraction (0.0s) - got '2026-10-05'
[PASS] graceful_empty_input (0.0s) - got {'error': 'no input provided'}

== agent v2 ==
3/4 passed
[FAIL] graceful_empty_input (0.0s) - raised ValueError: empty input not supported
[PASS] median_of_five_values (0.0s) - got 3, expected 3.0
[PASS] top_category_by_sales (0.0s) - got 'rice', expected 'rice'
[PASS] deadline_extraction (0.0s) - got '2026-10-05'

Both versions score 3/4. The headline number did not move, and that is the point: the score alone would have told you nothing. The diff tells you everything. One behavior fixed (median_of_five_values), one behavior regressed (graceful_empty_input). If you had hand-tested only the median fix, you would have shipped version 2 feeling good about it. The harness makes the tradeoff visible before your users do.

Diff the saved JSON files across versions and this becomes a one-line review step in your workflow: diff eval_results_v1.json eval_results_v2.json. Any line that flips from PASS to FAIL blocks the merge. That rule is worth more than the harness itself.

	Hand-testing prompts	Eval harness
What gets tested	The thing you just changed	Every behavior in the case set
Repeatability	Depends on your memory	Identical inputs, identical checks, every run
Regression detection	Only if you remember to look	Automatic, via the results diff
Failure evidence	"It looked wrong"	Named case + detail string + timing
Cost per change	Grows with your anxiety	One command, seconds
Honest failure mode	"I tested it, trust me"	A FAIL line with a name on it

The verdict:

Build the harness before you need it, which means now, while your agent is small. Four cases today become forty by Christmas, and the forty will include the one adversarial input that saves a production incident. The code above is the whole foundation. The expensive part was never the code. It is the habit of running it before every change and refusing to merge on a new FAIL.

Key takeaways

1. Hand-testing checks the change; a harness checks everything else. Every agent edit can move behavior you were not looking at. The case set is how you look at all of it at once.
2. Checks return (passed, detail), never just a boolean. The detail string is your 2 a.m. debugging head start.
3. The harness must survive agent crashes. An exception is data, not a harness failure. Record it as a FAIL with the exception type.
4. Score the diff, not the headline number. Both agent versions above scored 3/4. The regression only showed in the case-level diff, which is why results get saved as JSON and compared across versions.
5. Block merges on new FAILs. The harness is decoration until a red line can stop a deploy.

Do you have a fixed case set for your agent, or are you still testing by hand and hoping?

Related on Everyday Data Science: Agent Reflection Does Not Beat a Retry. The Numbers From 16,946 Trials. (yesterday's research brief on why matched baselines matter) and Building Multi-Agent Pipelines with LangGraph (the hands-on companion for when your agent grows past a single function).

Sources: VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets, Yu Bai et al., arXiv

.12404 (2026); all harness code and outputs in this tutorial were written and executed by the author for this piece.