The missing oysters: unverified tool output when every component passes its own tests
This blog is part of a series on common failure modes of agents. How do they occur despite following best practices? And why do they often go unnoticed? In this edition: unverified tool output, where the agent takes a tool result at face value without verifying it against available information. With potentially expensive consequences.
Fake news in agents
Fake news has become ubiquitous in politics and social media over the last decade (and arguably, it’s never not been there). Unexpectedly, agents are also exposed to fake news. Not in the sense that their LLM must interpret and validate fake news, but in that agents can take the input they receive as true at face value.
Unverified tool output is a prime example: a tool accessible to an agent returns incorrect information, and the agent acts on it without checking it against the other data it has. The agent takes the tool result at face value, despite having a source of truth it could verify against.
The agent simply forgets to verify the output, or never considered verification in the first place. Alerts on easy signals like tool errors and HTTP codes miss implicit failures, and reactive trace analysis breaks down as agents get more autonomous (just like in our previous blog on goal misinterpretation). On top of that, where real-world fake news often has a harmful intent, incorrect tool output is often neutral, which makes it even harder to spot.
Tools increase agent autonomy and usefulness, and are a necessity for most agents. But they also increase the risk of compromising agentic outcomes, and with them customer or user trust.
The oyster order at Valentine’s
To illustrate this, we built a synthetic restaurant ordering agent that places weekly supply orders. The restaurant has 118 covers booked for Saturday. The week before Valentine’s Day, the agent asks a forecasting tool how many oysters the Saturday Valentine’s serving needs. Before that call, the agent had already pulled the booked covers and the recipe for the Valentine’s set menu (3 oysters per plate), which was heavily promoted to customers.
The forecasting tool returns a forecast of 170 oysters. The agent orders 200, because oysters ship in cases of 100. One week later, on Valentine’s Day, the restaurant quickly runs out of oysters.
To illustrate the reasoning, here is a simplified version of the trace:
The tool and the agent weren’t wrong, per se
On a normal Saturday, the restaurant serves fewer oysters per plate. The forecasting tool was never given the Valentine’s set menu.
Evals and checks on the tool passed, including a parametric check that tests whether the output falls within a range given the inputs and the parameters of the meal. The tool was given a normal Saturday, so its output is in range.
On the agent’s side, it faithfully uses what the tool said and implicitly relies on the tool’s quality assurance measures. Every step in the agent trace looks correct, evals and checks on the LLM calls come back green, and there’s no tool error or failed LLM call.
No alerts fire, because there are no explicit errors.
The agent outcome was clearly wrong…
Running out of oysters that were heavily promoted is not just awkward, but also harmful for a restaurant. A negative Google review lives on forever.
The agent already held the evidence against the forecast: the booked covers and the set-menu recipe, three oysters per plate. Nothing checked the forecast against them. Every component passes its own tests; the failure hides in the seam between tool and LLM.
This is a different case from our previous blog, in which the agent misread its own goal. Unverified tool output is about trust between different parts of the system. Trust can already be an issue in a single-user, single-agent system, but it is amplified in multi-user, multi-agent systems.
…and it got noticed only when it was too late
You can try to avoid these errors ahead of time. You could explicitly test that the forecast takes events into account, or that the applied events correspond with the calendar. But it’s nearly impossible to address every idiosyncrasy, or to rule out false positives that unnecessarily restrict the agent, and definitely not without production data. On top of that, you end up with a long tail of small corrections to the basic forecasting flow, which can lead to a combinatorial explosion.
You can also create a lead agent or sub-agent that checks the forecast against the state of the ordering agent. But that adds complexity and can fail in its own distinct ways, not least because the multi-agent setup amplifies trust issues (as described above).
Simply using a more capable model doesn’t guarantee better use of tools either, as evidenced by Vals AI. How the model and its tools interact shapes the agent’s overall performance, not the model in a vacuum.
So maybe catch these implicit errors as soon as they occur instead. But while evals and checks on tools and LLMs are established, evals on the interaction between tool and LLM, the agent’s trajectory, aren’t. One-off testing and evaluation is insufficient to catch all possible failures, and at best happens a while after the error occurs. Checking each trace individually over-triggers, producing many false positives that erode the agent builder’s trust and add latency and cost.
What can you do?
This blog isn’t just scaremongering. In fact, addressing these issues does start with analysing your own traces for this pattern. For an initial take, it can be as simple as asking a coding agent: “find all of the implicit failures related to unverified tool outputs in my agent traces”.
But to mitigate this in production, you have to continuously monitor these traces and keep improving recursively, for example by setting up a persistent agent. Just prompting your coding agent will lead to inconsistent analyses over time, so it’s important to tune it. Once monitoring is in place, it produces insights that let you create effective but sufficiently precise tests, evals and checks, and iterate on them constantly.
It also gives you room for error. For the reasons above, the goal shouldn’t be to catch everything ahead of time. When you catch issues quickly enough, you can protect the customer experience while allowing the agent the autonomy that makes it more effective.
By doing so, you add a level of reliability that enables faster development of more autonomous and capable agents, and new types of products and user experiences.
If you’re curious how Moyai catches unverified tool outputs, we’ve created a demo for this specific example.
Agents look fine in demos. Production disagrees.
Moyai helps teams catch reliability issues early with clustering, evaluation and actionable alerts — so a failure surfaces before a customer or a regulator finds it for you.
If that sounds like the kind of tooling you want to run — try Moyai or join us on Discord.
We build Moyai for teams running AI in production: practical tools, fast feedback loops, and reliability you can operate.