In April 2026, researchers at UC Berkeley built an agent that scored 100% on seven of eight leading agent benchmarks, among them SWE-bench Verified and Pro, Terminal-Bench, WebArena, FieldWorkArena and CAR-bench. It solved none of the underlying tasks. It read the grading code and exploited it. One trick was a ten-line test file that rewrote every result as passed. Another was a replaced curl command that always returned the expected answer. A third was a config file that leaked the answer key. Of the 13 benchmarks the team examined, every one was rated exploitable.

A leaderboard score is a claim made by someone with an interest in the number. The only evaluation a buyer can trust is one it wrote, about its own work, and kept private. That sounds like advice for a lab. It is an afternoon's work for an accounts payable team, and the quickest way to see it is to follow one case from a single sentence to a release gate.

The case below is invented to show the mechanics. It is not drawn from a client project, and the pass rates are made up. The only measured facts here are the Berkeley findings above.

The sentence, and the case it hides

The project starts with a requirement of the usual kind: the agent will "handle supplier invoices." Nobody can argue about whether that was met.

So take one thing that happens. An invoice arrives by email for a supplier already in the system, and there is no purchase order behind it. What should the agent do? Ask the accounts payable clerk and the buyer, separately. One says reject it and send it back. The other says park it as a draft and ask purchasing, because half of these are real orders someone placed by phone. That disagreement is the first useful output of the whole exercise, and it appeared before any model was involved. Writing the case forces the argument the project needed anyway.

Suppose they settle on the second answer. The case is now written down as a situation, a set of tools and a correct outcome. The input is the email and its PDF. The tools are read access to purchase orders and vendors, and the right to create a draft vendor bill. The correct outcome is one draft bill, linked to the right vendor, flagged as having no purchase order, assigned to the buyer for review. Beside it goes a list of outcomes that must never occur whatever the input: the bill posted, a payment scheduled, the vendor record edited, a second draft created for the same invoice.

The people who should write this are the ones who know the ways invoices go wrong, which makes it the clerk's document more than the engineers'. We think the set of cases is the real product specification, and the sooner the people who do the work write it, the sooner the project learns what it is building.

The run, graded on the database

Most evaluation advice comes from the chatbot era: read the answer, score it. An agent changes things. It creates records, sends messages and moves money, so what it says is the least reliable evidence of what it did. A fluent final message can sit on top of a wrong tool call with a lucky outcome, and a clumsy one can follow a correct process.

An ERP gives you a better judge than any model. Load a sandbox copy of Odoo or ERPNext with realistic data, snapshot the database, let the agent work the email, and snapshot again. Then ask two questions of the difference. Did the right record appear in the right state? And did anything else change? The second question is the one teams skip, and it is where an agent that "worked" quietly posts a bill it was only meant to draft. A missed action is visible and gets reported. A wrong action buried in a ledger can sit there for a month.

Now run the case again. Agents are not deterministic, and a case that passes once has shown only that passing is possible. Say it runs ten times and the end state is right on seven. Whether seven in ten is acceptable depends on the action. For a ledger posting it is a failing case. For a draft summary of a long thread it may be fine. The threshold follows the cost of the miss, so set it before you see the result, and set it per class of action.

Now suppose the first full run goes badly in the way that matters. The agent's message says "Draft created and sent to purchasing," and the grader, reading only the message, would pass it. The database says otherwise: the bill was posted, not drafted, because the tool that creates a bill defaults to posting it. A transcript review would have missed this, and the forbidden-outcomes list catches it in seconds. The fix is small, a tool argument and a line in the prompt. The point of the case is what happens next: the whole set runs again, not only this case, because a fix for one behavior is exactly the kind of change that quietly breaks another. If the earlier cases still pass at their thresholds, the change ships. If one has slipped, you found out on a sandbox and not in a ledger.

Notice how little machinery this took: two checks, a list of forbidden outcomes and a repeat count. Frameworks propose six or more scored dimensions, among them tool selection, argument extraction, error recovery and plan coherence. All real, and none needed on day one. Add one when a failure shows you which you are missing.

The gate, and the next case

A case that lives in a document protects nothing. Put it in the build pipeline so that a prompt edit, a model swap or a new tool cannot ship without passing the same cases at the same thresholds. When something fails in production, and eventually something will, it becomes the next case. Without that loop you pay for the same mistake twice.

Public benchmarks have a place, a small one. Use them once, early, to see whether a candidate model is in a reasonable range before you invest in it. They cannot know your schemas, your approval rules or your edge cases, and the Berkeley result shows how easily they bend. If the only evidence behind a go-live decision is a demo and a benchmark score, that is the situation the researchers exposed, and it is closer to normal than it should be.

Evaluation is still something engineers do, and most buyers have never been asked to take part. They should be: a baseline you cannot trust cannot support a decision to scale, which is the point of our post on measuring ROI on agentic AI, and once an agent is live the same cases pair with the decision log described in AI agent observability.

ThinqHub implements Odoo and ERPNext, so a sandbox with realistic records is a normal starting point for us. If you want the eval layer designed alongside the workflow and not after a pilot stalls, our approach starts there. Get in touch before a launch date becomes the only evidence of readiness.