Rubrics
A rubric is a plain-English scoring guide you write for one agent. An LLM judge applies it to a sample of that agent's finished runs and records a score from 0 to 1 with a one-sentence reason. Use rubrics for things you cannot express as a check: tone, helpfulness, faithfulness to a source, whether the answer actually addressed the question.
Rubrics are separate from evaluations. Evaluations ask the same questions of every agent (did the run achieve its goal, how good was the output, what went wrong) and come out of your monthly allowance. Rubrics ask your questions, of the agents you choose, at the sample rate you set. Starter and above.
Writing a rubric
Rubrics live on the agent page in the dashboard. Each one has:
| Field | What it does |
|---|---|
| Name | Shows on the run and in alerts, for example Cites a source |
| Judge prompt | The scoring guide, in plain English. The judge sees it alongside the run's output and metadata, and scores each sampled run from 0 to 1: how likely the run meets it |
| Sample rate | The share of finished runs the judge scores, from 0 (off) to 1 (every run). Default 0.1 |
| Pass threshold | Optional. The score below which a run counts as failing this rubric. Needed for alerts |
| Alert on fail | Off by default. With a pass threshold and a fail streak, sends a rubric.failing event to your alert channels |
Write the rubric the way you would brief a careful reviewer. Say what a good answer contains and what a bad one does, and name the things that matter most:
Score this support reply on three things:
1. It answers the question the customer actually asked.
2. It names the next step and who takes it.
3. It does not promise a refund, credit or timeline.
A reply that fails point 3 scores no higher than 0.3 whatever else it does well.
The judge responds with a score between 0 and 1 (1 fully satisfies the rubric) and a short reason. Both are stored on the run and shown in the dashboard, so a low score always comes with the why.
Sampling
The sample rate is the cost dial. Sampling is deterministic per run, so retries and replays make the same choice and a run is never judged twice against the same rubric version. 0.1 on a high-volume agent gives plenty of signal; set 1 on a low-volume agent you care about.
Versioning
Every edit to the rubric text creates a new version (rubv_<region>_...). Scores are tagged with the version they were produced against, so changing a rubric does not rewrite the history of scores before the change. The agent page shows the current version; earlier scores keep pointing at the version that produced them.
Alerts
With a pass threshold set and alerting on, a run scoring below the threshold counts as a failure. The alert fires once the fail streak is reached and once per failing episode, then rubric.recovered closes it when a run scores above the line again. Streak semantics match check alerts.
Cost
Rubric judging runs on the same decision model as Verify's evaluations, on the sample you chose. There is no separate charge today and it does not come out of your evaluation allowance. Start with deterministic checks for anything expressible as a rule, and keep the sample rate low on busy agents.