Guide

Monitor your LlamaIndex agents and RAG pipelines.

A LlamaIndex query engine or agent retrieves, reasons and calls the model. AgentPing registers as a handler so one record lands per query with the cost, status and answer quality behind it.

Track your first AI workflow Jump to setup

Two lines, then every run reports in.

Install the SDK and point it at your LlamaIndex agent. One record lands per run with the cost, status, latency and output behind it.

shellinstall
pip install agentping
pythonmonitor your agent
import agentping
from agentping import AgentPingLlamaIndexHandler
from llama_index.core import Settings

agentping.init(api_key="apk_…", agent="docs-rag")
Settings.callback_manager.add_handler(AgentPingLlamaIndexHandler())

# Query as usual; retrieval and model calls are captured per query.
response = query_engine.query("…")

The handler captures retrieval and model calls under one run per query, so cost and quality are tracked together. Full reference is in the docs.

What AgentPing tracks for LlamaIndex.

  • Cost per query, across retrieval and the model calls.
  • A missed-run alert when a scheduled pipeline stops running.
  • Answer quality, scored against a rubric (cites a source, answers the question, stays on policy).

See the pillars behind these: Spend, Pulse and Verify.

Does it cover RAG retrieval too?
Yes. The handler captures the full query, including retrieval and the model calls, as one run, so you see cost and answer quality together.
Can it catch a bad RAG answer?
Yes. Write a rubric in plain English (cites a real source, answers the question) and AgentPing scores production answers against it, so a confident wrong answer is flagged.
Do I rewrite my query engine?
No. Register the handler on the callback manager once; your query code stays the same.

Monitor your LlamaIndex agents in minutes.

Add the SDK, send one run, and watch cost, status and quality land in the dashboard.

Track your first AI workflow All monitoring guides