How evaluations work

An evaluation is an AI review of whether a run achieved its goal. There is nothing to configure: no scorers to write, no prompts to tune. Give your runs a goal (the SDK's goal field, or agentping.goal on OpenTelemetry traces) and AgentPing reviews the outcome.

Two layers run on every account:

  • Health checks: automatic, on every run, on every plan, no limits. Did it complete, did it error, was it unusually slow or expensive, did a tool spin in a loop, did it produce output.
  • Evaluations: the AI review. Each plan includes a monthly allowance: 1,000 on Starter, 6,000 on Team, 20,000 on Business.

How your allowance is spent

  • Spread evenly across the month. Evaluations pace themselves so coverage is steady rather than spent in the first week.
  • Smart sampling. Routine healthy runs don't use up your allowance: when an agent keeps doing the same thing and keeps passing, AgentPing samples it lightly and concentrates evaluations on new and unusual runs. These runs show as covered by routine baseline.
  • Multi-agent fairness. When you have multiple agents, no single agent can use more than half your allowance.
  • Returned when we fail. If an evaluation fails on our side, it is returned to your allowance automatically.

Tell us when we get it wrong

See a result you disagree with? Tell us with one tap on the run's evaluation panel. It directly improves how we evaluate your runs.

When the allowance runs out

Health checks continue, unlimited. Evaluations pause until next month, a top-up, or an upgrade. Nothing else changes.

Per-team control

Each team has an "Evaluate runs" switch. Teams you create later (a staging team, a client team) start with it off, so experiments never consume your allowance unless you choose.

FAQ

Why wasn't this run evaluated? Either its team has evaluations switched off, it was covered by the routine baseline, or the month's allowance was already spent at the time.

Can I evaluate every run? If your volume is near your allowance, top-ups cover a busy month; if it is consistently above, the next tier is cheaper.

Do top-ups roll over? No; they apply to the current month only.

How long do results take? Most evaluations appear within the hour; the run shows "Evaluation in progress" until then.