The uncomfortable version of this post is one sentence. Your eval suite is a photograph of the traffic you had on the day you wrote it, and you are using it to make claims about traffic you have not seen.
Most teams know this in the abstract. Fewer have looked at how fast the gap opens, which in my experience is faster than people expect and in a direction that is nearly invisible from inside CI.
How the gap opens
An eval set gets written at a moment. Somebody exports a few hundred representative inputs, labels the expected behaviour, and wires it into CI. On that day it is an excellent proxy for production, because it literally was production.
Then the product moves. A new customer segment arrives with different phrasing. Marketing changes the onboarding copy and the questions people ask change with it. A feature ships that routes a new category of input into the same agent. Someone integrates a partner whose data has different formatting conventions. None of these are unusual and none of them touch the eval set.
Six months later the suite is still testing the July distribution faithfully and reporting a healthy number, while a meaningful slice of live traffic looks nothing like anything in it. The suite is not lying. It is answering a question about the past very accurately, and everyone is reading it as an answer about the present.
The failure mode this produces is specific and nasty: the agent gets worse only on the inputs the suite does not contain. That is the definition of undetectable by that suite, and it is why "our evals pass" can coexist with a support inbox that is quietly getting worse.
Model changes make it worse, silently
The other half of this is that the thing under test moves too.
Providers update models behind aliases. Sometimes that is announced, sometimes it is a quiet behavioural shift within a version that was supposedly pinned. Either way the effect is that your agent's behaviour changes on a day you did not deploy anything.
An eval suite catches this only if the change happens to affect the cases in the suite. Because model updates tend to shift behaviour on ambiguous or unusual inputs before they shift it on clean ones, and eval sets tend to be built from clean representative examples, the suite is systematically biased against noticing. The cases most likely to move are the ones least likely to be in it.
This is the mechanism behind most of the quality drift stories I hear, and it is why "we pinned the model version" is a weaker guarantee than it sounds.
Measuring the gap directly
The measurement is simpler than the problem suggests. Run your rubric against a random sample of live runs, and plot that score next to your eval score.
def sample_and_score(run, rate=0.03):
if random.random() > rate:
return
verdict = judge.invoke(rubric_prompt(run["input"], run["output"]))
emit_score({
"run_id": run["id"],
"score": verdict["score"],
"source": "production",
})
Same rubric on both sides, which is the part people get wrong. If the eval suite grades against expected outputs and the production sampler grades against a quality rubric, the two numbers are not comparable and the gap is meaningless. Use one rubric for both, even if it means loosening the eval side.
Then read the divergence rather than the levels. Eval at 0.91 and production at 0.88 is a healthy pair. Eval at 0.91 and production drifting from 0.88 to 0.74 over three weeks is the finding, and the finding is not really "quality dropped", it is "the suite no longer represents the traffic". Those call for different responses.
Three percent sampling is usually enough. You want a direction over days, not a confident estimate on any one day, and the marginal value of a bigger sample falls off quickly against a cost that does not. The mechanics of running a judge over live output, including where it goes wrong, are in LLM-as-judge in production.
Why a bigger eval set is not the fix
The instinct on seeing a gap is to expand the suite. Add the failures, broaden the coverage, get from 90 cases to 900.
Worth doing, and it does not solve the problem. A bigger photograph is still a photograph. You have made the sample more representative of the traffic you have seen, which does nothing about the traffic arriving next quarter, and the gap reopens at the same rate as before from a slightly better starting point.
There is also a practical drag. Large eval suites get slow and expensive to run, which pushes teams to run them less often, which is the opposite of what you want. And a suite that takes eleven minutes in CI gets skipped under deadline pressure.
The honest division of labour is that the eval set catches regressions you can anticipate, and continuous sampling catches the ones you cannot. They are complements. Neither replaces the other, and a team with only one of them has a blind spot in a predictable place.
What good looks like
Four things, none of them large:
A suite that runs in CI on the cases you know matter, kept deliberately small enough to stay fast.
Sampled scoring on live traffic with the same rubric, running continuously, on a trend line somebody actually looks at.
A standing habit of promoting production failures into the suite when you find them, which is how the suite stays useful rather than becoming archaeology.
Deploy markers on the production quality chart, so a step change has a candidate cause attached to it rather than requiring an investigation to place in time.
That last one sounds trivial and is the difference between noticing a regression in two days and noticing it in five weeks. A line that steps down on the same day as a release is nearly self-diagnosing. The same line without markers is just noise until somebody thinks to correlate it.
The broader framing of measuring agents once they are live, rather than before, is in how to evaluate AI agents in production, and writing a rubric that survives the model upgrades this post is worried about is covered in how to write an LLM rubric that survives a model upgrade. If you want the production half of this without building a scoring pipeline first, Verify does the sampling and trend line, and it is free to try on one agent.