The agent in this walkthrough is a research assistant. It takes a question, decides whether it needs to search, optionally searches, drafts an answer, critiques its own draft, and either revises or returns. Five nodes, one conditional edge, one loop back. It is deliberately ordinary, because the interesting part is not the graph.
It ran in production for two months with no instrumentation beyond application logs. This is what it took to make it answerable, in the order the questions came up, including the piece I built first and would now skip.
The graph, before
from langgraph.graph import StateGraph, END
from langchain_anthropic import ChatAnthropic
llm = ChatAnthropic(model="claude-sonnet-5")
def needs_search(state): ...
def search(state): ...
def draft(state): ...
def critique(state): ...
def revise(state): ...
graph = StateGraph(ResearchState)
graph.add_node("triage", needs_search)
graph.add_node("search", search)
graph.add_node("draft", draft)
graph.add_node("critique", critique)
graph.add_node("revise", revise)
graph.set_entry_point("triage")
graph.add_conditional_edges("critique", lambda s: "revise" if s["needs_revision"] else END)
app = graph.compile()
Four of the five nodes call a model. The loop back from critique to revise can run more than once. Nothing anywhere records what any of it costs.
Start with the accumulator, not the tracer
My instinct was to wrap every node in a decorator that emitted a span. I did build that, and I have since taken most of it out, for reasons I will come back to.
The thing worth building first is smaller: an accumulator that rides along in graph state and collects usage as nodes execute.
from typing import TypedDict, Annotated
import operator
class ResearchState(TypedDict):
question: str
draft: str
needs_revision: bool
# accumulates across every node, including loop repeats
usage: Annotated[list[dict], operator.add]
Declaring usage with operator.add matters more than it looks. LangGraph merges state updates from each node, and the default merge replaces. With operator.add the lists concatenate instead, which means a node that runs three times through the revision loop contributes three entries rather than overwriting the previous two. The first version of this quietly undercounted every multi-pass run, and undercounting is worse than not measuring, because it produces a number people trust.
Each node then returns its usage alongside its real output:
def draft(state: ResearchState) -> dict:
response = llm.invoke(build_draft_prompt(state["question"]))
meta = response.usage_metadata
return {
"draft": response.content,
"usage": [{
"node": "draft",
"input_tokens": meta["input_tokens"],
"output_tokens": meta["output_tokens"],
"cache_read": meta.get("input_token_details", {}).get("cache_read", 0),
}],
}
usage_metadata is already on the response. No extra call, no wrapper, no measurable overhead. Pull the cache-read figure out separately while you are here, because cached input bills at a fraction of the normal rate and a cost calculation that ignores it will overstate spend badly on any agent with a long stable system prompt. Prompt caching covers why that gap gets large.
One event at the end
With usage accumulating in state, the whole run collapses into a single structured event:
import time, httpx
def run_agent(question: str, customer_id: str) -> str:
started = time.time()
outcome = "success"
try:
final = app.invoke({"question": question, "usage": []})
answer = final["draft"]
except Exception:
outcome = "error"
answer = ""
raise
finally:
emit({
"agent": "research-assistant",
"customer_id": customer_id,
"outcome": outcome,
"duration_ms": int((time.time() - started) * 1000),
"passes": sum(1 for u in final.get("usage", []) if u["node"] == "revise") + 1,
"cost_by_node": summarise(final.get("usage", [])),
"cost_total": total_cost(final.get("usage", [])),
})
return answer
Two details that were not obvious to me at the start.
The emit belongs in finally, so failed runs are recorded too. An observability system that only sees successes will tell you your agent is perfect, and the runs you most want to look at are exactly the ones that threw.
customer_id goes on the event because it can never be added later. Once the run is over, the association between this spend and this customer exists nowhere else, and it is the dimension every awkward question turns out to need. The same applies to any feature or tenant identifier you have to hand.
passes is specific to this graph and is the field I have got the most out of. It counts revision loops, and the distribution of that number tells you more about the agent's health than the cost figure does. A day where mean passes drifts from 1.3 to 2.1 is a day something changed upstream, and it shows up before the cost chart looks alarming.
Freshness comes free
The event carries a timestamp, so the highest-value alert in the whole exercise is now available at no extra cost: tell me when this agent has not reported a run in longer than it should.
This agent runs on user demand rather than a schedule, so the threshold is a quiet-hours-aware baseline rather than a fixed window. For anything cron-driven the rule is simpler and stricter, and cron monitoring best practices for scheduled AI agents has the specifics. Either way this is the alert that catches the failure mode with no upper bound on duration, which makes it the one I would wire first if I could only have one.
The quality verdict
Cost and freshness tell you the agent ran and what it cost. Neither tells you whether the answer was any good, and for this agent the failure that matters is a plausible answer that is subtly unsupported by the sources.
The critique node already produces a judgment, so the temptation is to reuse it as the quality score. Do not. A model grading its own output in the same run, with the same context, is measuring its own confidence rather than its correctness, and it is confidently wrong in exactly the cases you need to catch.
What works is a separate scored pass on a sample of runs, with a rubric, out of band:
def score_sample(event, rate=0.05):
if random.random() > rate:
return
verdict = judge.invoke(rubric_prompt(event["question"], event["answer"]))
emit_score({"run_id": event["run_id"], "score": verdict["score"], "reason": verdict["reason"]})
Five percent is plenty for a trend line, and a trend line is what you want. Individual scores are noisy and mostly not worth reading. The aggregate moving down over three days after a deploy is the signal, and it is the one that catches a quality regression nobody reports. Writing a rubric that holds up across model upgrades is its own skill, covered in how to write an LLM rubric that survives a model upgrade.
The part I would skip
The per-node span tracing I built first. It was a decorator on every node emitting start and end events with the full state at each boundary.
It produced roughly forty events per run against the one the summary approach produces. It was genuinely useful for about a week, while I was debugging why the conditional edge routed strangely on short questions. After that I read approximately none of it, and it was the largest single contributor to ingest volume.
The distinction that took me too long to see is that step-level tracing is a debugging tool and per-run summary is a monitoring tool. They look similar and they are not. Debugging is bursty, deliberate, and short-lived. Monitoring is continuous and cheap. Running a debugging tool continuously is how observability bills end up rivalling inference bills, which is a bad trade for data nobody reads. There is more on where the boundary sits in OpenTelemetry for AI agents.
Turn the tracing on when you have a question. Leave the summary on always.
Where this ends up
Four fields per run, cost, outcome, timestamp, customer, plus a sampled score and one node-count field specific to this graph. That is enough to answer what it costs, who it costs it on, whether it is running, and whether it is still any good.
It took an afternoon, most of which was the accumulator merge bug. The quickstart covers the same pattern in TypeScript and Go if LangGraph is not your stack, and you can point it at one agent for free to see what a month of your own runs looks like before deciding what else is worth recording.