These are composites. The details are real, drawn from teams running agents in production over roughly the last month, but identifying specifics are changed and some incidents merge two similar cases. I have written them up because the pattern across all four is more useful than any one of them, and because the pattern is boring in a way that is genuinely good news.
None of the four failures threw an exception. All four were found by a human, late. And in each case the signal that would have caught it was already being produced by the agent, just not recorded anywhere.
The scheduler that stopped
A nightly enrichment agent ran at 2am, pulled the day's new accounts, called a model to categorise each one, and wrote results back. It had run every night for four months.
Then a dependency upgrade changed the signature of the job's entry point. The scheduler kept firing, the process started, and it exited almost immediately with status 0. No exception. No alert. The queue was empty every morning because nothing was ever queued.
Eleven days later a salesperson asked why the new accounts in her territory had no category on them. Eleven days of accounts, roughly 4,000 records, all needing a backfill that then cost more than a fortnight of normal running, because backfills run without the natural rate limiting of a trickle.
The tell was available on night one: the agent produced zero runs, where it had produced between 300 and 900 every night for four months. Freshness monitoring is the least glamorous thing in observability and it catches the most expensive class of failure, because a silent agent is the only failure mode with no natural upper bound on how long it can persist. We wrote about the general shape in how to catch a silent AI agent before your customer does, and the fix is genuinely one alert: tell me if this agent has not reported a run in 90 minutes.
The retry storm
A support-triage agent handled about 12,000 tickets a month at a stable and unremarkable cost. A release added a structured-output requirement, and a validation step that retried when the model returned malformed JSON.
The retry had no cap. On the roughly 2% of tickets containing an attachment with unusual encoding, the model returned something that failed validation every time, and the agent retried until the process timed out. Each retry resent the full conversation, which by attempt twelve was substantial. A 2% slice of traffic became most of the bill.
What made this expensive was not the bug, which is a one-line fix. It was that the shared provider key meant the dashboard showed one aggregate line going up. Three engineers spent the better part of two days reconstructing which agent was responsible from raw logs, and they only started looking because the monthly invoice had roughly tripled. Thirty-three days between the deploy and the diagnosis.
Per-agent daily spend would have made this a thirty-second question. The chart for that one agent departs from its own baseline on the day of the deploy and never comes back. We took a similar case apart run by run in anatomy of an £8,900 token bill. The pattern is common enough that if you take one thing from this post, make it a cap on retries and a per-agent view of what you are spending.
The regression nobody reported
This is the one I find most uncomfortable, because the team did almost everything right.
A content-classification agent had an eval suite. Ninety-three cases, run in CI, passing. In late June the provider silently updated the underlying model behind a version alias the team had pinned, or believed they had pinned. Outputs shifted. The eval suite still passed, because the eval set had been written eight months earlier against the kind of content the product had then.
Nobody filed a bug. Classification is the sort of task where being wrong 15% of the time instead of 4% does not produce complaints, it produces a slow drift in downstream metrics that gets attributed to seasonality. It surfaced five weeks later when someone ran a manual audit for an unrelated reason.
A passing eval suite is evidence that your agent still handles the cases you thought of when you wrote it. It is not evidence that it handles today's traffic. Sampling live runs and scoring them continuously is a different measurement, and it is the one that moves when the world changes underneath you. That distinction is the whole subject of the gap between your eval set and your users, and the mechanics of scoring live output are in LLM-as-judge in production.
The tool call that worked
The last one is different in kind, and it is the failure mode I expect to hear more about as agents get more autonomy.
An internal operations agent could call a small set of tools, including one that issued account credits. A prompt change intended to make it more helpful about billing questions made it more willing to resolve them. It began issuing credits for complaints that warranted an explanation instead.
Every one of those tool calls succeeded. Valid parameters, 200 response, credit applied, audit log written. There is no technical signal anywhere in that chain that says anything is wrong, because nothing is wrong technically. The agent did exactly what it was asked. It was asked for the wrong thing.
This found its way out through finance, not engineering, eight days and a few thousand pounds later.
I do not think there is a clean monitoring answer to this one, and I would be suspicious of anyone selling one. What helps is narrower: track the rate of consequential tool calls per hundred runs as a metric in its own right, and alert on the rate rather than on any individual call. A credit-issuing rate that quadruples after a deploy is visible in a way that no single credit ever is. The general principle, that the actions an agent takes deserve the same scrutiny as the text it produces, is one we go into in the production-readiness checklist for AI agents.
What the four have in common
Detection time was set by how close a human sat to the output. The credit agent had finance downstream, so eight days. The triage agent had an invoice, so thirty-three. The classifier had nobody, so five weeks, and it was luck that ended it. Severity had nothing to do with it.
The other thing they share is more encouraging. Each incident was already visible in data the agent was generating at the time. The scheduler knew it had run zero times. The triage agent knew what each retry cost. The classifier had its outputs. The ops agent had its tool calls. Nothing needed to be inferred or reconstructed, only recorded and watched.
That is why the honest starting point is not a tracing architecture. It is one event per run with a cost, a verdict, and a timestamp, which between them cover three of the four incidents above and put a floor under the fourth. AgentPing is built around that event, so you can start with a single agent for free and pick the one you would least like to find out about from a salesperson in eleven days.