Verify alerts

Quality failures do not throw. A run that returns confident nonsense is a 200 to every other tool you own. Verify alerts page you when checks fail and when rubric scores slip, with streak controls so one flaky run does not wake anyone.

Check alerts

Each check has two alert settings:

• Alert on fail: off by default; turn it on per check.
• Fail streak: how many consecutive runs must fail before the alert fires. Default is 1, meaning the first failing run alerts.

When the streak is reached, every enabled channel receives one check.failing event listing the failing checks with their failure detail. The alert fires once per failing episode; you are not re-paged while the check stays red. When the check passes again, a single check.recovered event closes the loop and the streak counter resets.

Rubric score alerts

Rubrics scored by LLM-as-judge add a third setting:

• Pass threshold: the score below which a run counts as failing that rubric.
• Alert on fail and fail streak: same semantics as checks.

A rubric.failing event carries the score, the threshold, and the judge's reasoning for the low score, so the alert itself usually tells you what changed. Recovery sends rubric.recovered once the score is back above the line.

Streaks are the tool for tuning noise. A customer-facing agent with a tight rubric might warrant streak 1; a noisy judge on low-stakes output is happier at 3.

Quality drift

Drift is the slide that never crosses a hard threshold: each run scores a little worse, and the trend only shows in aggregate. The Verify dashboard watches the trailing 14-day daily average judge score and flags when the latest day breaks materially below that baseline, surfacing a drift banner with the affected window.

Today drift is a dashboard signal; delivery of drift events to alert channels is in development. If you want a pager-grade guard on quality in the meantime, a rubric pass threshold with a fail streak is the sharp tool: it catches the same regressions one run at a time.

For writing rubrics the judge can score consistently, see Rubrics.