Percentile latency is one of the most useful ideas in operating web services, and it transfers to agents worse than almost anything else we brought with us.
The assumption underneath p99 is that requests are comparable. A hundred requests to the same endpoint do roughly the same work, so if one of them takes ten times as long, something is wrong with that request rather than different about it. That assumption is what makes the number meaningful, and agents break it on purpose.
Variable work is the whole point
An agent run might make one model call or nine. It might search, or decide it does not need to. It might loop through a critique step twice because the first draft was weak. Those are not performance variations, they are the agent doing its job, and they produce a latency distribution with enormous natural spread.
So a p99 of 47 seconds against a p50 of 6 tells you almost nothing on its own. It might mean 1% of runs are stuck. It might mean 1% of your inputs are hard and correctly get more work. Those two situations want completely different responses and the metric cannot separate them.
Worse, the number is stable in the face of things you would want to know about. If your agent starts needing three passes instead of two on a fifth of traffic, that is a real regression with a real cost attached, and it can move p99 hardly at all while moving the median and your bill quite a lot.
The timeout problem
This is the part that turns a weak metric into an actively misleading one.
Runs that time out generally do not appear in your latency data. Either they are classified as errors and excluded from latency aggregation, or, more commonly, the instrumentation emits its event on the success path and a run that dies never emits at all.
The slowest runs are the likeliest to time out. So the mechanism systematically deletes the worst observations from the metric. When your agent degrades badly enough to push more runs past the timeout, p99 goes down, because the tail got truncated rather than because anything improved.
I have watched a team celebrate a latency improvement that was entirely this. The fix is unglamorous and total: emit the event in a finally block with the duration and the outcome, so a timeout is a recorded run with a long duration and a failure verdict rather than an absence.
started = time.time()
outcome = "success"
try:
result = agent.run(payload)
except TimeoutError:
outcome = "timeout"
raise
finally:
emit({
"agent": "research-assistant",
"outcome": outcome,
"duration_ms": int((time.time() - started) * 1000),
"model_calls": counter.value,
})
Once timeouts are in the data, the tail tells the truth, and it usually looks considerably worse than it did.
Two series instead of one
The replacement is not a cleverer percentile. It is splitting the question in two, because a total duration is the product of two independent things.
Work per run. How many model calls, tool invocations, or graph passes a run needed. This is a distribution worth watching in its own right, and it moves for interesting reasons: input quality changing, a tool returning worse results so the agent retries, a prompt change making the model less decisive.
Duration per unit of work. How long each call took. This is the number that behaves like classical latency, because each model call genuinely is comparable to another, and this is where provider slowness and network problems show up cleanly.
Total duration rising with flat per-call latency means the agent is doing more work. Total duration rising with flat work per run means something outside your agent got slower. The total alone cannot distinguish those, and they have nothing in common as problems.
That first series is the one nobody instruments and the one that pays. In the LangGraph walkthrough the single most useful field turned out to be a count of revision passes, which is exactly this: a work-per-run measure that moved days before anything else did.
What to alert on
Not p99. Three things instead.
Runs that produced no outcome at all. The timeout and hang class, counted as a rate rather than a latency. Any nonzero value here is worth knowing about, and it is invisible in a percentile by construction.
Work per run departing from its own baseline. A step change in mean model calls per run is one of the earliest and most reliable indicators that something upstream changed. It is often the first thing to move.
Per-call duration against the provider's own behaviour. Useful because it distinguishes your problem from theirs, which is worth knowing before you spend a day optimising something you do not control.
Latency still matters for anything with a person waiting, and I am not arguing you should stop measuring it. I am arguing that for agents it is a derived quantity, and that watching the derived quantity while ignoring its two factors is how you end up with a dashboard that stays green through a regression. The wider set of things traditional monitoring was not built to see is in beyond Sentry and Datadog, and the failure that hides best of all, an agent that stops producing runs entirely, is in how to catch a silent AI agent.
AgentPing records duration, outcome, and work per run on the same event, including for runs that failed, which is mostly a matter of putting the emit in the right place. Try it on one agent and look at what your tail actually contains.