Your LLM output is getting worse. The dashboard says green.

A prompt change last Monday, a model auto-upgrade, a system prompt edit. Something shifted, and the answers your agent returns have been quietly degrading since. Production tells you everything is healthy. Support is filling up with tickets that sound similar. By the time someone bisects, the change is two weeks old.

The output looks fine. Until you compare it to last week.

LLM output is non-deterministic, so any single response could be a one-off. A small quality drop on a single run is indistinguishable from normal variance. Drift only shows up in the aggregate: a moving average dipping below threshold, a series of low-scoring runs sharing the same failure mode, a distribution that has subtly shifted left.

Where the signal is and isn't

  • Latency dashboards. Useless for quality. A degraded answer arrives in the same 800ms as a good one.
  • Error rate. Useless for quality. The agent didn't throw; it confidently returned the wrong thing.
  • Offline evals. Run once at deploy time, never again. Production traffic has a different distribution from your test set, and that's where drift lives.
  • Customer complaints. The most reliable signal, and the latest. By the time tickets arrive, the regression has been shipping output to everyone for days.

Score every run, watch the distribution, page on drift.

Two layers of scoring catch drift between them. Deterministic checks on every run, free, at zero LLM cost. LLM-as-judge on a sampled subset for the qualitative dimensions (tone, correctness, completeness) that deterministic checks can't reach.

A real rubric

support-triage · Correct triage · v3 rubric 
judge prompt: |
  Score this support ticket triage on three things:
  1. The department is the right one for the issue.
  2. The severity matches how urgent the customer is.
  3. The customer's tone is preserved in the summary.
  A triage that gets the department wrong scores no
  higher than 0.3 whatever else it does well.
judge model: claude-haiku-4-5
sample rate: 0.10            # 10% of finished runs
pass threshold: 0.8
alert on fail: 3 in a row    # rubric.failing to Slack

The rubric is plain English, with the scale pinned in the prompt so the judge has fixed reference points. Edit and save; new runs score against v4, historical scores stay tagged to v3, and the judge's one-line reasoning is stored with every score.

The three things to instrument

  • Deterministic checks on every output. JSON schema, regex, length bounds, required fields, tool-call assertions, numeric range. Six primitives, zero LLM cost, catch the shape regressions before they reach a customer.
  • LLM-as-judge on a sampled subset. Plain-English rubric, a sample rate you set per rubric, a cheap judge model by default. Catches qualitative drift that deterministic checks can't see.
  • Drift detection on the score distribution. z-score against the trailing 14-day baseline. A single 2/5 is noise; the mean dropping from 4.2 to 3.8 over a week is signal. The drop is flagged on the dashboard with the low-scoring runs and the judge's reasoning.
My evals pass in CI. Why do I need scoring in production?
Offline evals run on a fixed test set. Production scoring runs on real user input distribution, which drifts over time. A prompt change can pass every CI test and still produce worse output on the long tail of inputs you didn't test. Production scoring is the only place that distribution lives.
Doesn't LLM-as-judge cost a lot?
Only if you score every run. Rubrics run on a sample: 10% of an agent's runs by default, any rate from 0 to 1 per rubric, chosen deterministically so retries don't get scored twice. The judge is also a cheap model: Claude Haiku 4.5 by default. Cost is bounded by the sample rate; the signal is not.
How do you keep judge scores comparable as the rubric evolves?
Two things. Each rubric edit creates a new version and scores are tagged with the version that generated them, so a rubric change shows up as a break in the chart rather than a mysterious drift. And the rubric itself should pin the scale: describe what a 0.2 and a 0.9 look like inside the prompt, and the judge holds to those descriptions across versions.
What's the difference between deterministic checks and LLM-as-judge?
Deterministic checks (JSON schema validation, regex, length bounds, required fields, tool-call assertions, numeric range) run on every output at zero LLM cost. They answer pass/fail questions about shape. LLM-as-judge answers quality questions about content (tone, correctness, completeness) on a sampled subset. Most teams run both layers.
How does drift detection avoid flapping on single bad runs?
Drift runs on the distribution, not single scores. A single 2/5 is noise. The mean moving from 4.2 to 3.8 over a week is signal. Statistical change detection (z-score against the trailing 14-day baseline) flags it on the dashboard, with the low-scoring runs and the judge's reasoning, so you see the root cause, not just the symptom.

How Agent Ping implements quality scoring.

Verify is the quality-scoring side of Agent Ping. The features page walks through rubrics, the sample rate dial, and drift detection thresholds. The docs go one level deeper.

Verify features → Verify docs → What is AI agent observability? →