OpenTelemetry is the closest thing the industry has to a neutral way of getting telemetry out of an application. For AI agents it matters more than usual, because agents are built on a dozen different frameworks and nobody wants a separate proprietary SDK for each one. If your framework speaks OTel, your monitoring can be framework-agnostic. Here is the honest scope of what that buys you.
What OTel gives you
A standard schema. The GenAI semantic conventions define attributes for model requests and responses: the model name, input and output tokens, the operation, and spans for agent steps and tool calls. So a span from LangChain and a span from Semantic Kernel describe the same things in the same words. That is the whole point of a convention.
A standard transport. OTLP is the wire format. You export spans to a collector, or straight to a backend that accepts OTLP, over HTTP or gRPC. No bespoke ingest per tool.
Framework reach without bespoke code. Many agent frameworks already emit OpenTelemetry, or have an instrumentation package that does. When they do, you get traces of agent runs without writing capture code yourself.
This is genuinely useful. A trace of a run, with each model call and tool call as a span and token counts on each, is most of the raw material you need to understand what happened.
Where OTel stops
OTel is a pipe and a schema. It deliberately does not decide what the data means operationally. Three gaps show up immediately for agents:
- The run as a unit. A trace is a tree of spans. "This run cost $0.04, succeeded, and scored 3.8" is a roll-up the backend has to compute. OTel carries the spans; it does not define the run-level record you actually operate on.
- Cost. Token counts are in the spans, but a price is not. Turning tokens into money, cache aware, from a maintained rate card per provider and model, is backend work. A raw trace tells you 8,000 tokens, not 4.3 cents on the right cost centre.
- Absence and quality. OTel records what happened. It cannot tell you that a scheduled agent should have run at 14:00 and did not, or that a successful run produced a worse answer than last week. Schedule freshness and output scoring are judgements layered on top of the data, not properties of the data.
None of this is a criticism of OTel. It is doing its job. But "we emit OpenTelemetry" and "we can operate our agents" are different sentences.
Turning traces into answers
The pattern that works: let your framework emit OTel, and send it to a backend that reconstructs the run and adds the operational layer. Concretely, the backend should:
- Assemble the run from the spans, so cost, latency, status and tools roll up per run and per agent.
- Price it server-side from a rate card, cache aware, so the number matches the invoice rather than a client guess.
- Watch for absence by comparing actual runs against the schedule each agent is supposed to keep, and page when one goes missing.
- Score the output with deterministic checks and a judged sample, so a confident wrong answer is caught.
Do that and the trace stops being a debugging artifact you open after an incident, and becomes a live operational signal.
AgentPing accepts OTLP directly: point your existing OpenTelemetry exporter at it and the agent run is reconstructed, priced, schedule-checked and quality-scored, no second instrumentation. See the OpenTelemetry docs, or get started free.