LlamaIndex

LlamaIndex reports what a query engine, chat engine or agent does through its callback manager. AgentPing ships a handler for Python (llama-index-core); register it once on Settings and every model call, embedding batch, retrieval and exception inside LlamaIndex lands on the active run. Python only.

Install / enable

pip install "agentping-io[llamaindex]"

import agentping
from agentping import AgentPingLlamaIndexHandler
from llama_index.core import Settings, VectorStoreIndex
from llama_index.core.callbacks import CallbackManager

agentping.init()
Settings.callback_manager = CallbackManager([AgentPingLlamaIndexHandler()])

index = VectorStoreIndex.from_documents(documents)
engine = index.as_query_engine()

with agentping.run("kb-answer", customer_id="acme-corp"):
    answer = engine.query("What is our refund window?")

AgentPingLlamaIndexHandler(run=None). Supported llama-index-core versions: 0.10 up to, but not including, 2.0. Set the handler on Settings.callback_manager for global coverage, or pass a CallbackManager to a specific index, engine or LLM to scope it.

Events

LlamaIndex event AgentPing event What is recorded
CBEventType.LLM llm_call Provider, model, input and output tokens as the underlying LLM reports them in response.raw, cached input tokens when OpenAI reports them, latency.
CBEventType.EMBEDDING llm_call with kind: "embedding" Provider, model, latency, and an estimated input_tokens (chunk characters divided by four, since LlamaIndex does not surface embedding usage). output_tokens is 0.
CBEventType.RETRIEVE retrieve node_count: how many nodes the retriever returned.
CBEventType.EXCEPTION error The exception message and class. The run stays open; finish it with the status you want.

Model name comes from the event payload's model_name (or the serialised LLM's model / deployment_name), then from the response. provider is inferred from the model name: claude-* is anthropic, gpt-*, o1, o3 and text-embedding-* are openai, gemini-* is gemini, mistral-* and mixtral are mistral, command-* and embed-* are cohere, and anything else is llamaindex. Pass a model name the rate card recognises and cost follows.

Honest limits. Token counts are taken as the provider returned them, so an Anthropic call lands with the net input count and its cache split is not applied; use the Anthropic wrapper alongside the handler if that matters for your spend numbers. Embedding token counts are an estimate. Tool calls inside a LlamaIndex agent and the agent's own steps emit nothing; record them yourself with run.event("tool_call", {...}) from inside the tool function.

Naming and runs

The handler reads the active run from the current context, so create it once at startup and wrap each unit of work in with agentping.run(...). Events that fire outside a run are dropped. An explicit run= is honoured for retrieve and error events; llm_call events go to the active run, so keep LlamaIndex calls inside the run's with block.

Source / notes

For span-level traces of a whole query (retriever, synthesiser, each LLM call with timing), OpenInference's openinference-instrumentation-llama-index exports LlamaIndex traces over OTLP; point it at the OpenTelemetry endpoint.