The unauthorised refund: detecting goal misinterpretation when your traces look clean
This blog is part of a series on common failure modes of agents. How do they occur despite following best practices? And why do they often go unnoticed? In this edition: goal misinterpretation, where the agent misunderstands the intent and optimises for the wrong objective. With potentially expensive consequences.
Imagine your airline uses a customer service agent. A customer requests a refund on a basic economy ticket, bought weeks ago without insurance. Per policy, the agent is not authorised to issue a refund, because the ticket is non-refundable. You have guarded the agent against tool errors and hallucinations.
We built a synthetic, Tau3-bench-inspired airline customer service agent to reproduce this scenario. It shows how goal misinterpretation can make your agent fail with direct financial impact, despite your best efforts and despite following best practices for agent governance.
The agent consists of a policy document, booking tools, and a simulated customer on the other end. The setup is based on an article from Vals AI [1] on how far personal AI assistants will go to get a refund on a non-refundable ticket. We use this example because the impact is tangible and quantifiable, but similar situations can occur with agents at insurers, lenders, legal firms and elsewhere.
Goal misinterpretation and its traces
To show how goal misinterpretation appears in an agent trace, we annotated and simplified a pseudo-trace of a failed agent run, based on a realistic sample of failed synthetic traces.
The trace shows that, although the agent interprets its policy correctly, it misinterprets its overall goal and looks for multiple ways to get the customer an unauthorised refund. Eventually, the agent adds insurance to the booking retroactively, which allows an insurance claim on the cancelled booking. In effect, it creates a refund.
Where the goal got sidetracked
Our example agent got its goal wrong through a series of interconnected problems that compound. It starts correctly: the agent checks its judgement against its own policy. The policy uses a soft refusal, which lets the agent keep talking to the customer. It shouldn’t freeze, even when a request conflicts with the policy, so a hard stop is not a rule or fix that can be applied here.
After that, however, the agent re-reads its task and stops applying the policy. Or at least, it misinterprets how the policy applies to the session goal, which in its eyes is getting the customer a refund. Given that goal, upgrading the fare or adding insurance are logical next steps.
Why your checks came back clean
To understand these issues, look at where traditional monitoring falls short. Traditional monitoring of agent traces spots single, isolated, explicit errors: tool mistakes, false claims, or missed constraints. It can’t show the implicit failures that come from the agent misinterpreting its goal.
Each line on its own shows nothing wrong. The agent read the policy. Every tool call succeeded. The customer presumably left happy. Look at all the actions together, however, and the pattern is wrong.
You can find isolated errors with a specific rule or a structured query, which points you to the single step or span in the trace where the error happens. But goal misinterpretation doesn’t happen in a single step or span. The failure spans many steps, tools and model calls. It’s the relationship between the decisions, and the reasoning that connects them, that shows how the agent fails.
Even then, not every refund request leads to goal misinterpretation, which hides the failure from view and makes it even harder to prevent ahead of time. Nor did the customer ask for an upgrade or insurance, which would have pointed to a more isolated adversarial behaviour such as prompt injection.
Why ad hoc trace analysis won’t help
Right now, we can only spot this failure if a customer or internal user reports that the agent made a mistake. The traces then help engineers investigate the incident. They can see what the agent did and the task it was given, and use their findings about incorrect actions to change the setup.
This whack-a-mole approach misses many failures. The analysis happens long after the failure. It only produces isolated improvements to the agent. And analyses done one after another can reach inconsistent results.
It also creates new problems. Reading every trace takes time. Working out whether a trace span is correct takes significant research, as does finding the root cause of an issue and defining a solution. That makes the process slow and expensive.
A reviewer without full context can also overreact. Hand a chargeback trace to an LLM and it flags it as “risky” based on the topic alone. This produces many false positives, which in turn causes alert fatigue, and the cycle comes full circle with newly missed issues.
What to check instead
Most importantly, connect the individual spans and analyse the session as a whole. Check whether the actions connect to the request. Look at the policy behind them: for each sensitive action in the session, did someone request it? Were they authorised to do so?
In our example, the agent upgrades the fare and adds insurance. Neither relates to what the customer actually asked for, and the policy doesn’t clearly list either as an allowed action.
Second, agent tracing isn’t a set-and-forget task, which makes it different from traditional monitoring. The implicit failures are in the monitoring data, but you can only detect them by analysing that data. Combine monitoring and analysis properly and you don’t need to reach full reliability before launching. You get there by improving the agent in production: keep making changes and reduce the long tail of failures to the level you need. That lets you validate your product and ideas faster.
Where does your agent say no in situations like this? And how might the tools it uses lead it to misunderstand its true goal?
References
- Vals AI, “The impossible refund.” Read the article on the Vals AI blog.
Agents look fine in demos. Production disagrees.
Moyai helps teams catch reliability issues early with clustering, evaluation and actionable alerts — so a failure surfaces before a customer or a regulator finds it for you.
If that sounds like the kind of tooling you want to run — try Moyai or join us on Discord.
We build Moyai for teams running AI in production: practical tools, fast feedback loops, and reliability you can operate.