Plain-English definitions of the words Agent Ping uses across the dashboard, docs, and SDK. Each term has a stable anchor so you can link to a definition directly. If something here is missing, tell us and we'll add it.
A single invocation of an AI agent, captured as one telemetry record. The unit of analysis for cost attribution, monitoring, and quality scoring.
A run may make zero or many LLM calls; it is still one record.
Tracing AI spend back to the agent, team, customer, feature or model that generated it, by tagging each run as it starts and pricing it from a rate card. The provider invoice reports spend by API key; attribution adds the dimensions the business actually uses.
The full definition, and the two layers it can happen at: what is AI cost attribution?
A client-generated UUIDv7 identifier for an agent run. Constructed locally before any network call so the agent has an ID immediately and idempotent retries work.
Format: run_ prefix plus 32 hex characters.
A server-side mapping from (provider, model) to per-token prices. Agent Ping prices every run from the rate card so the SDK never sends a cost number.
Defaults ship for Anthropic, OpenAI, Gemini, xAI, Mistral, DeepSeek, Cohere and TypeSafe (Jev) models (see LLM API pricing); override per team.
Tokens served from a provider-side prompt cache. Anthropic prompt-cache reads and OpenAI cached_tokens are priced separately from fresh input.
Cache creation is billed as regular input; cache hits are usually 90% cheaper.
Total cost divided by the number of runs that finished cleanly (status=success and quality threshold met). The number finance can build budgets around.
Excluding failed and timed-out runs avoids hiding loops in the average.
Per-agent statistical check on today's spend against a rolling 14-day baseline. Today's spend more than 2.5σ above the trailing mean fires the agent's alert route.
Threshold configurable per team; default catches most retry loops on day one.
A one-call signal sent at the end of a scheduled job, indicating "I ran". The cheapest form of monitoring for systems that cannot host a full SDK.
Agent Ping accepts heartbeats at GET /v1/ping and POST /v1/heartbeats.
A monitoring pattern that pages when a scheduled agent does not run inside its expected window. Set a cron expression plus a tolerance window per agent.
Distinct from APM exceptions: the signal is the absence of a run, not an error.
The grace period after a scheduled run's expected time before the alert fires. Prevents flapping on normal queue jitter.
Default 5 minutes; configurable per agent.
Five-field schedule string (minute, hour, day of month, month, day of week) telling Agent Ping when a scheduled agent should run.
Agent Ping also accepts plain-English shorthands like "@daily" or "every 5 minutes".
The 95th-percentile run duration over a rolling window (24h, 7d, 30d). Most runs land below it; the upper tail is where regressions hide.
A sudden p95 move usually means a model swap or a longer prompt.
A plain-English scoring guide for an agent's output. The judge model reads the rubric and produces a numeric score with reasoning.
Created on the agent page with a judge model, a sample rate, an optional pass threshold and alerting. Every edit is versioned.
The LLM that scores production runs against a rubric. Agent Ping defaults to Claude Haiku 4.5; configurable per rubric.
Haiku, Sonnet or Opus per rubric. The sample rate is the cost dial.
The technique of scoring one LLM's output by asking another LLM to evaluate it against a rubric. Sample-based, so the judge costs a fraction of the agent itself.
Catches qualitative drift that deterministic checks cannot see.
A non-LLM validator that runs on every output at zero LLM cost: JSON schema, regex, length bounds, required fields, tool-call assertions, numeric range.
Answers pass/fail questions about shape, not quality.
Concrete descriptions of what each score level looks like, written into the rubric itself, so a judge pins its scale to fixed text rather than its own sense of good and bad.
A practice, not a setting: put the anchors in the rubric prompt and keep a hand-scored set to replay when the rubric or judge model changes.
The fraction of an agent's finished runs scored against a rubric. Sampling is deterministic per run, so retries and replays make the same choice.
Default 0.10 (10%); set 0 to pause a rubric and 1 to score every run.
The score, on the rubric's 0 to 1 scale, that a run must reach to count as passing that rubric.
Optional per rubric; required to alert on failing scores.
Every rubric edit creates a new version. Scores are tagged with the version that generated them, so changes do not retroactively re-score history.
The agent page shows the current version; earlier scores keep pointing at the version that produced them.
A statistical change-detection signal on the score distribution. A single 2/5 is noise; the mean dropping from 4.2 to 3.8 over a week is drift.
z-score against the trailing 14-day baseline.
When agent A spawns agent B, A is the parent and B is the child. Agent Ping surfaces the full tree so a downstream failure can be traced to its root run.
Propagated via env var AGENTPING_PARENT_RUN or explicit parameter.
A full-access team credential, prefixed apk_. Used by the SDK in header authentication.
A leaked API key compromises the whole team; rotate immediately.
An open-source observability platform for AI agents, built around session replay: a Python-first SDK records each agent session so you can step back through its LLM calls, tool calls and errors, with cost tracking per session. Strongest for debugging a single run after the fact.
How it compares: AgentOps and Agent Ping, what each is for and AgentOps alternatives.
A per-agent scoped credential, prefixed ping_. Safe to embed in URLs for cron heartbeats and shell scripts.
A leaked ping token only compromises that one agent's heartbeat.