War stories, cost breakdowns, and the small amount of instrumentation that catches the failures we keep watching teams walk into. Notes from building Agent Ping.
Six additions in one release: version labels that mark deploys on the Spend and Verify charts, time to first token on Pulse, chat sessions stitched into one run over OpenTelemetry, a sortable fleet view, prompts and completions on the run timeline, and an investigation paragraph on every alert.
LangSmith charges per seat and per trace, with two retention tiers and a usage unit called the LSU. What each plan includes, what a trace is, and worked bills at 1M, 10M and 50M traces a month.
Langfuse Cloud bills by the unit, where a unit is a trace, an observation or a score, with graduated overage. What each plan includes, how an agent run becomes a dozen units, and worked bills at 1M, 10M and 50M units.
Braintrust meters processed data in gigabytes and scores in thousands, with unlimited users on every plan. What Starter and Pro include, how spans turn into gigabytes, and worked bills at 1M, 10M and 50M spans a month.
Datadog prices LLM Observability per 10,000 LLM spans, with a free tier, a Pro tier from $160 a month, and three billing modes. What the tiers include, what overage and retention cost, and worked bills at 1M, 10M and 50M spans.
AI cost attribution is the practice of tracing AI spend back to the agent, team, customer, feature and model that generated it. What it is, why the provider invoice cannot do it, the two places you can do it, and what a working setup needs.
A laravel/ai agent with a tool, instrumented in one composer require. What each step cost, what the tool was asked and what it returned, and what a failure looks like when the run itself reports success.
Connect Agent Ping to any MCP-compatible assistant and ask it what failed, what a run cost, what the evaluation said, and on paid plans have it acknowledge the incident and tune the rule. Reads are on every plan, including Free.
Four production failures from a single month, anonymised and composited: a silent scheduler, a retry storm, a quality regression nobody reported, and a tool call that succeeded at the wrong thing. Each one had a tell, and each tell was cheap to watch for.
The questions that decide whether your agent survives budget season are not technical, and they are surprisingly consistent. Here are the six that come up, what makes each one hard to answer from a provider dashboard, and the shape of a good answer.
A working LangGraph agent with no observability, and what it takes to make it answerable: cost per run, per-node attribution, freshness, and a quality verdict. With the code, and with the parts that turned out not to be worth it.
The closest tool to us, compared honestly. Where AgentOps is the better choice, where we are, and the architectural difference that explains most of the rest. Checked against both products in August 2026.
A passing eval suite tells you the agent still handles the cases you thought of when you wrote it. It says nothing about today's traffic. Here is how that gap opens, how to measure it, and why the fix is not a bigger eval set.
Percentile latency was designed for services where every request does the same amount of work. Agents violate that assumption completely, which is how a healthy-looking p99 hides a class of runs that never finish at all.
Multi-agent systems break cost attribution in a specific way: the spend lands on the sub-agent while the reason for it lives in the caller. Here is how to keep both, and why summing per-agent costs stops adding up to your bill.
ClickHouse acquired Langfuse in January. Nothing has broken, and that is exactly when migration decisions are cheap. Here is what changes after an infrastructure acquisition, what to check in your own usage, and the honest case for staying.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, for three reasons: escalating costs, unclear business value, and inadequate risk controls. Each one is an observability gap before it is a strategy problem. Here is how to close all three.
Putting an AI agent in production means deciding where its data lives, what leaves to model providers, and what your monitoring can see. Here are the questions a security review will ask, and how to answer them.
Total spend hides the truth about an AI agent. The number that does not is cost per successful run, because it counts the retries and loops that burned money for nothing. Here is how to think about agent unit economics.
Offline evals tell you a model is good on a fixed set. Production tells you whether it is still good on real traffic. Here are the layers of agent evaluation, from deterministic checks to LLM-as-judge, and how to run them without an eval pipeline.
OpenTelemetry can carry your agent telemetry, and the GenAI semantic conventions standardise it. Here is what OTel covers for AI agents, what it deliberately leaves to you, and how to turn traces into operational answers.
A straight comparison of the tools teams use to monitor AI agents in production, what each is built around, and when to reach for which. Covers Agent Ping, LangSmith, Helicone, Langfuse, Braintrust and Datadog.
Shipping an agent to production is not the same as shipping a service. The checklist that keeps a web service healthy misses the failure modes that hurt agents most. Here is the one that does not, grouped by the three questions you have to be able to answer.
Most AI cost advice is "use a smaller model and hope". These seven levers cut spend by changing how the agent works, not how well it works, and several of them are pure waste reduction with no trade-off at all.
Three acquisitions in five months. The LLM observability layer is consolidating into bigger infrastructure companies. Here is what the pattern means if you are picking an observability tool right now, and the architectural question every team has to answer.
Sentry is excellent at the job it was built for, and your agent will hit failure modes that job does not include. A bad answer is not an exception. Here is exactly where the line falls, and how to cover the other side of it.
A rubric is the definition of "good" for your agent. Write it loosely and your quality score drifts every time the model under it changes. Write it well and it becomes the most durable thing you own. Here is how to write one that holds.
Most agents send the same long system prompt on every single call and pay full price for it every time. Prompt caching can cut that portion of the bill by up to 90%. Here is how it works, when it pays, and how to tell whether you are leaving the money on the table.
Scheduled jobs fail by going quiet, and a quiet job produces no signal at all. The discipline that catches it is older than AI agents, but agents add two wrinkles. Here is the full practice, from heartbeat to grace window to alert route.
A monitoring tool that floods you on day one gets its channel muted, and a muted monitor is dead. Here is how Agent Ping alerting works now: one notification when something breaks, one when it recovers, and an Issues page that shows what is wrong right now in one place.
Using a model to score your model is the only way to grade nuance at scale. It is also a measurement instrument that can drift, flatter, and contradict itself. Here is how to run it so the scores mean something.
The provider dashboard gives you one number for the whole account. That number cannot answer the three questions you actually have. Here is how to break spend down by agent, by customer, and by feature, and what each breakdown unlocks.
A composite incident pieced together from three real teams. One agent, one retry loop, one ballooning context window, and a bill nobody saw coming until the first of the month. What the daily view would have shown, and on which day.
Silent breakage, mystery bills, and quality drift. The three failure modes that teams shipping AI agents keep walking into, and the smallest amount of instrumentation that catches each one on day one.
Scheduled agents fail differently. They go quiet. No exception, no 500, no paging signal. Heartbeats and expected cadences close the gap between "stopped running" and "we know about it".
Quality slid 22% over two weeks. Nothing crashed. The model upgrade had changed how the agent read its system prompt. Continuous scoring closes the gap between "we changed something" and "we know it got worse".
Traditional APM watches infrastructure. AI agents fail differently. A web service crashes loud; an agent keeps returning something. Usually plausible. Sometimes wrong. Always billable.