War stories, cost breakdowns, and the small amount of instrumentation that catches the failures we keep watching teams walk into. Notes from building AgentPing.
Putting an AI agent in production means deciding where its data lives, what leaves to model providers, and what your monitoring can see. Here are the questions a security review will ask, and how to answer them.
Total spend hides the truth about an AI agent. The number that does not is cost per successful run, because it counts the retries and loops that burned money for nothing. Here is how to think about agent unit economics.
Offline evals tell you a model is good on a fixed set. Production tells you whether it is still good on real traffic. Here are the layers of agent evaluation, from deterministic checks to LLM-as-judge, and how to run them without an eval pipeline.
OpenTelemetry can carry your agent telemetry, and the GenAI semantic conventions standardise it. Here is what OTel covers for AI agents, what it deliberately leaves to you, and how to turn traces into operational answers.
A straight comparison of the tools teams use to monitor AI agents in production, what each is built around, and when to reach for which. Covers AgentPing, LangSmith, Helicone, Langfuse, Braintrust and Datadog.
Shipping an agent to production is not the same as shipping a service. The checklist that keeps a web service healthy misses the failure modes that hurt agents most. Here is the one that does not, grouped by the three questions you have to be able to answer.
Most AI cost advice is "use a smaller model and hope". These seven levers cut spend by changing how the agent works, not how well it works, and several of them are pure waste reduction with no trade-off at all.
Three acquisitions in five months. The LLM observability layer is consolidating into bigger infrastructure companies. Here is what the pattern means if you are picking an observability tool right now, and the architectural question every team has to answer.
Sentry is excellent at the job it was built for, and your agent will hit failure modes that job does not include. A bad answer is not an exception. Here is exactly where the line falls, and how to cover the other side of it.
A rubric is the definition of "good" for your agent. Write it loosely and your quality score drifts every time the model under it changes. Write it well and it becomes the most durable thing you own. Here is how to write one that holds.
Most agents send the same long system prompt on every single call and pay full price for it every time. Prompt caching can cut that portion of the bill by up to 90%. Here is how it works, when it pays, and how to tell whether you are leaving the money on the table.
Scheduled jobs fail by going quiet, and a quiet job produces no signal at all. The discipline that catches it is older than AI agents, but agents add two wrinkles. Here is the full practice, from heartbeat to grace window to alert route.
A monitoring tool that floods you on day one gets its channel muted, and a muted monitor is dead. Here is how AgentPing alerting works now: one notification when something breaks, one when it recovers, and an Issues page that shows what is wrong right now in one place.
Using a model to score your model is the only way to grade nuance at scale. It is also a measurement instrument that can drift, flatter, and contradict itself. Here is how to run it so the scores mean something.
The provider dashboard gives you one number for the whole account. That number cannot answer the three questions you actually have. Here is how to break spend down by agent, by customer, and by feature, and what each breakdown unlocks.
A composite incident pieced together from three real teams. One agent, one retry loop, one ballooning context window, and a bill nobody saw coming until the first of the month. What the daily view would have shown, and on which day.
Silent breakage, mystery bills, and quality drift. The three failure modes that teams shipping AI agents keep walking into, and the smallest amount of instrumentation that catches each one on day one.
Scheduled agents fail differently. They go quiet. No exception, no 500, no paging signal. Heartbeats and expected cadences close the gap between "stopped running" and "we know about it".
Quality slid 22% over two weeks. Nothing crashed. The model upgrade had changed how the agent read its system prompt. Continuous scoring closes the gap between "we changed something" and "we know it got worse".
Traditional APM watches infrastructure. AI agents fail differently. A web service crashes loud; an agent keeps returning something. Usually plausible. Sometimes wrong. Always billable.