How to write an LLM rubric that survives a model upgrade

A rubric is the definition of "good" for your agent. Write it loosely and your quality score drifts every time the model under it changes. Write it well and it becomes the most durable thing you own. Here is how to write one that holds.

Matt King

June 23, 2026 10 min read

A rubric is the most durable thing you will write for an agent. The prompt will change, the model will change, the framework will change. The definition of what "good output" means for this agent, if you write it well, outlives all of them. Write it badly and the opposite happens: your quality score becomes a thing that moves every time anything underneath it moves, and you can never tell whether the agent got worse or the measurement did.

This is how to write one that holds. It pairs with LLM-as-judge in production, which covers running the judge; this covers the thing the judge reads.

A rubric is a definition, not a vibe

Start by being honest about what most "quality scores" actually are. They are a model being asked "is this good, 1 to 5", with no shared definition of good. That produces a number, the number goes on a chart, the chart looks rigorous, and it measures almost nothing stable, because "good" is doing all the work and "good" means something slightly different to every judge and every model version.

A rubric replaces the vibe with a definition. It names the criteria that actually matter for this specific agent, weights them by importance, and pins each point on the scale to a concrete description. The shift is from "is this good" to "did it do the three specific things that make this agent's output good", which is a question you can answer the same way twice.

Criteria: name what actually matters

Good criteria are domain-specific, concrete, and few. For a support-triage agent, "quality" is meaningless, but these are not:

Routing accuracy. Did it send the ticket to the correct queue per the queue charter?
Priority assignment. Is the priority within the SLA matrix, and how far off if not?
Customer summary. Does the summary capture the actual question and its context?

Each of those is something a judge can assess against an external standard rather than its own taste. The test for a criterion is repeatability: if the same output could plausibly score a 2 or a 5 depending on the judge's mood, the criterion is too loose and needs to be sharpened until it has a real target.

Keep the list short. Three to five weighted criteria is usually right. A rubric with fifteen criteria is not more rigorous; it is harder for the judge to apply consistently and harder for you to reason about when a score moves. Weight the criteria so the most important one dominates; routing accuracy at 0.5 and customer summary at 0.2 says, correctly, that sending the ticket to the right place matters more than the prose.

Anchors: the thing that survives a model upgrade

Here is the failure this whole post is named after. You ship a rubric that says "routing accuracy: score 1 to 5". It works. Months later the provider updates your judge model, and your routing-accuracy scores shift by half a point with no change to the agent. You spend a morning hunting a regression that does not exist, because the thing that moved was the judge's interpretation of an undefined scale.

Anchoring fixes it. Do not leave the judge to decide what a 3 means. Tell it:

routing_accuracy:
  weight: 0.5
  1: "Routed to a clearly wrong queue (different department)."
  3: "Routed to an adjacent but suboptimal queue."
  5: "Routed to the exact correct queue per the queue charter."
priority_assignment:
  weight: 0.3
  1: "Priority off by two levels (a P3 marked P1)."
  3: "Priority off by one level."
  5: "Priority correct per the SLA matrix."
customer_summary:
  weight: 0.2
  1: "Summary misses the customer's actual question."
  3: "Summary captures the question but loses key context."
  5: "Summary captures question and context faithfully."

Now a "1" is pinned to a concrete description, not the model's free-floating sense of badness. When the judge model changes, the anchors hold the scale in place, because the reference points are fixed text rather than the model's discretion. Anchored scales are the single highest-leverage thing in this entire practice, and they cost you nothing but the few minutes it takes to write what each score looks like.

Calibration: prove the rubric measures your intent

A rubric is a hypothesis: "this captures what I mean by good". You validate it the same way you would any measurement instrument, against a known standard.

Build a calibration set of 30 to 50 outputs you have hand-scored, spanning the full range from genuinely bad to genuinely excellent. Run your rubric and judge over them and compare the machine scores to your scores. Strong correlation means the rubric captures your intent and you can trust it on live traffic. Divergence means it is measuring something other than what you meant, and you fix it now, before it is grading thousands of real outputs.

The calibration set keeps paying off. Replay it whenever you change the rubric or the judge model, and the delta tells you whether you changed the instrument or the thing being measured. A rubric you have calibrated is one you can defend; a rubric you have not is a confident number of unknown meaning.

Versioning: never re-score the past silently

Rubrics change, and that is fine, as long as the change is visible. The rule is that a run is scored against the rubric version that was live when it ran, and that score is immutable.

When you update the rubric, runs from that point forward are scored against the new version, and the chart marks the boundary with a vertical line. Now a step in the trend is attributable: it lines up with a rubric version change, so you know the measurement moved, not the agent. Without rubric versioning, a quality "improvement" the week you tightened the rubric is indistinguishable from the agent actually getting better, and conflating those two is exactly the confusion the whole exercise is meant to remove.

If you genuinely want old runs scored against the new rubric, do it as an explicit backfill that is visible on the chart, never as a silent overwrite that quietly rewrites history.

A worked rubric, end to end

Putting it together, a production rubric for the triage agent looks like this:

agent: support-triage
version: 3
scale: 1-5
criteria:
  routing_accuracy: { weight: 0.5, ... anchors above ... }
  priority_assignment: { weight: 0.3, ... }
  customer_summary: { weight: 0.2, ... }
calibration_set: triage_calibration_v3.jsonl

The judge sees the original ticket, the agent's output, and this rubric. It returns a weighted score and a one-line rationale per criterion. The score lands on the run, the rationale lands in the event timeline, and the dashboard plots the distribution with the version marker in place. When something drifts, the criterion-level rationales tell you which dimension slipped first, which is usually the fastest route to the cause. See Verify features for how this runs.

The prompt is disposable; you will rewrite it a dozen times. The rubric, written well, is the stable definition of quality that lets you tell whether each rewrite helped. That is why it is worth the hour. Get started and write the rubric for your most customer-facing agent before you ship the next prompt change.

What is an LLM rubric?

A rubric is the explicit definition of what good output looks like for a specific agent, written so that a judge (deterministic checks, an LLM judge, or both) can apply it consistently. It names the criteria that matter, weights them, and describes what each score on the scale actually means. It is the difference between asking "is this good" and asking "does this route to the correct queue, assign the right priority, and summarise the customer faithfully", which is a question that can be answered the same way twice.

How specific should rubric criteria be?

Specific enough that two reasonable people scoring the same output would land on the same number. Vague criteria like "quality" or "helpfulness" invite drift because every judge interprets them differently. Concrete criteria tied to your domain (routed to the correct queue per the charter, priority within one level of the SLA matrix, no fabricated facts) give the judge fixed targets. The test is repeatability: if the same output can plausibly score a 2 or a 5 depending on mood, the criterion is too loose.

Why do rubrics drift when the model changes?

A loose rubric leans on the judge model's own sense of quality to fill in the gaps, and that sense shifts between model versions. When the provider updates the judge model, a rubric that said only "score 1 to 5 for quality" gets interpreted slightly differently, so your scores move even though your agent did not change. Anchoring each score to a concrete description removes the judge's discretion and keeps the scale fixed across model versions.

Should I version my rubrics?

Yes, always. A run should be scored against the rubric version that was active when it ran, and that score should be immutable. When you change the rubric, runs from that point are scored against the new version and the chart marks where the version flipped, so a step in the trend is attributable to the rubric change rather than confused with a change in the agent. Re-scoring old runs against a new rubric should be an explicit, visible backfill, never a silent overwrite.

How do I validate that a rubric is good before trusting it?

Build a calibration set: 30 to 50 outputs you have hand-scored, spanning genuinely bad to genuinely excellent. Run the rubric and judge against them and compare the machine scores to your hand scores. If they correlate well, the rubric captures what you mean by quality. If they diverge, the rubric is measuring something other than your intent, and you fix it before it goes near live traffic. Replay the same set whenever you change the rubric or the judge model.