Why AI Agent Evals Go Stale (and How to Tell)
Evals rot — but almost never for the reason people assume. An eval doesn't decay because it's old on the calendar. It decays because the frontier of agent capability moved past what the eval can still tell apart. A test that once cleanly separated a strong agent from a weak one becomes a formality when every current agent passes it, and a threat model written for last year's agents can miss how this year's fail. Measuring eval currency means asking whether a pack still discriminates on the current frontier — not how long ago it was written.
Quick answer: An AI eval goes stale when the frontier of agent capability advances past what it can still discriminate, not when it reaches some calendar age. You measure currency two oracle-free ways: recency — how many dated frontier cohorts behind the current one the eval was last validated against — and headroom — how much score spread it still produces among current frontier agents. A stale eval isn't broken and doesn't fail; it's out of date, and it's capped out of a top grade until it's revalidated.
Why do AI evals go stale?
Two forces, both about capability rather than time:
- Saturation. As agents improve, more of them clear the bar. An eval where every current agent scores near the ceiling has stopped separating good from great — it still runs, it just no longer discriminates.
- Shifting failure modes. New agent architectures fail in new ways. A threat model or rubric tuned to how last generation's agents broke can be blind to how the current one does.
Neither of these is measured by publication date. That's the core mistake in treating "old eval" as "bad eval": age is at best a loose proxy for the thing that actually causes decay.
How do you measure whether an eval is still current — without an oracle?
There's no oracle for "the best eval," so currency, like every axis of a credible verification authority, is scored in ways that need no ground truth. Two of them:
| Measure | What it tracks | What "stale" looks like |
|---|---|---|
| Recency | How many dated frontier cohorts behind the current one the pack was last validated against | It hasn't been re-run since the frontier moved on |
| Headroom / saturation | The score spread the pack still produces among a cohort of current frontier agents | Every current agent clusters near the ceiling |
Recency is the load-bearing measure today: an eval validated against a cohort several generations old is flagged regardless of its calendar date. Headroom is the natural complement — it catches saturation directly — but it only activates once real dated frontier cohorts are seeded to measure spread against, so it's best understood as the second half of the picture that recency anchors now. The currency benchmark ranks every pack on this axis.
What happens to a stale eval — does it fail?
No — and this distinction matters. Currency is a lifecycle state, not a smoothly additive score. A pack moves through states as it falls behind the frontier:
| State | Meaning | Consequence |
|---|---|---|
| Fresh | Validated against the current frontier cohort | No cap |
| Aging | A cohort or so behind, still discriminating | Watch; no cap yet |
| Stale | Behind enough that discrimination is at risk | Capped out of a top grade until revalidated |
| Expired | Far enough behind that its verdicts aren't trustworthy | Pulled from sale pending review |
A stale eval that measured the right thing correctly is out of date, not broken — so it drops a grade rather than failing outright. This is different from a structurally defective eval (one that leaks answers or can be gamed), which fails on its own terms. Currency demotes; it doesn't condemn. For what a genuine structural failure looks like instead, see how to detect a gameable eval.
How is this different from just re-running the eval every year?
A yearly re-run is calendar-driven and measures nothing about capability. The currency approach is frontier-driven: it revalidates when a new dated cohort of agents appears, because that's the event that actually threatens the eval's ability to discriminate.
It's also where the durable advantage lives. Anyone can copy a pack's test cases. What they can't copy is a maintained series of dated frontier snapshots to revalidate against — the cohorts are what let you say whether a pack is still current at all. A cloned eval goes stale the moment the frontier moves, and only whoever keeps the cohorts can tell. The measuring stick, not the stock of tests, is the thing that compounds.
Related
- What is a reference-panel harness — the panel method that currency revalidates over time.
- How to detect a gameable eval — a structural failure, contrasted with staleness above.
- How to eval a RAG agent's groundedness — one capability whose packs are held to this currency axis.
- Capability packs — packs graded on currency alongside the other axes.
- The medical RAG groundedness pack — a verified pack carrying a currency state.
- Currency benchmark — how every pack ranks on the currency axis.
FAQ
No. Calendar age is a poor proxy for decay. An eval built two years ago that still cleanly separates today's best agents from broken ones is current; an eval written last month against an outdated cohort can already be stale. What matters is whether it still discriminates on the current frontier, not how long ago it was authored.
A wrong eval measures the wrong thing or measures it incorrectly — that's a structural defect, and it fails. A stale eval measured the right thing correctly, but the frontier has moved past what it can still tell apart. Stale is a lifecycle state, not a defect: it caps a pack out of a top grade until it's revalidated, rather than failing it outright. Out of date is not the same as broken.
On the cadence of the frontier, not the calendar. Revalidate when a new dated cohort of frontier agents lands, because that's when the eval's ability to discriminate is genuinely at risk — not on a fixed annual schedule. Tying revalidation to calendar time either wastes effort or misses a capability jump.
Yes. If a pack is built and validated only against an old cohort of agents, it ships already behind the frontier — new on the calendar, stale in substance. This is exactly why currency is measured against dated capability cohorts rather than the pack's publication date.
Headroom is the spread an eval still produces among a cohort of current frontier agents. If every current agent scores near the ceiling, the eval has saturated — it's become a formality that no longer separates good from great. Wide spread means the eval still has discriminating room left; a collapsed spread is an early warning that it's going stale.
Because age doesn't cause decay — a capability shift does, and the two are only loosely correlated. Measuring months-since-publication would flag healthy evals as stale and pass genuinely saturated ones. The oracle-free measures that actually track decay are how many frontier cohorts behind the eval was last validated, and how much spread it still produces among current agents.
They can copy the test cases; they can't copy the dated frontier snapshots the currency judgment rests on. Keeping a maintained series of capability cohorts to revalidate against is the part that compounds over time — the cloned pack goes stale the moment the frontier moves, and only whoever maintains the cohorts can tell.