How to Tell If Your AI Eval Is Contaminated (and Fix It)
An eval is only useful if a good agent passes and a bad one fails. Contamination quietly destroys that. Once the test cases leak — into training data, a blog post, or CI logs — a model can score well simply because it has seen them, not because it is any good. The eval keeps producing numbers, but the numbers stop meaning anything.
Quick answer: An eval is contaminated when a model scores well because it has seen the test cases — in training data, blogs, or CI logs — not because it is good. Detect it with a perturbation test: if a model passes the original but fails a trivially reworded variant, it memorized the answer. The fix is held-out, rotating cases you never ship.
What is eval contamination, and why does it break your measurement?
An eval's entire value is its discriminating power: the gap between what a good agent scores and what a broken one scores. Contamination collapses that gap. When cases are memorized, even a weak model recognizes the answer, so it scores like a strong one and the eval can no longer tell them apart. You are then flying blind while believing you have instruments.
How do you detect a contaminated eval?
Three practical checks, cheapest first:
- Perturbation test — take a case the model passes and reword it trivially (rename entities, change numbers, reorder clauses) without changing the underlying task. A model that reasons still passes; a model that memorized now fails. A large pass-to-fail gap on trivial edits is the clearest contamination signal.
- Canary strings — seed unique, unlikely tokens into your private cases. If those tokens later surface in model outputs or on the web, your set has leaked, and you can even attribute where.
- Anomaly check — if a frontier model suddenly aces a hard set with no visible reasoning, or a new model version jumps unrealistically, suspect that the set entered its training data.
Why do published evals decay fastest?
The more an eval is shared, the faster it dies. A benchmark posted publicly gets scraped, blogged, and folded into the next training run; a suite pasted into public CI logs leaks case by case. This is why a single "ultimate" public eval is self-defeating — the moment everyone tests against it, everyone optimizes to pass it rather than to be good, and its discriminating power evaporates. Plurality and privacy aren't a business trick; they're a requirement for the measurement to keep working.
What are held-out and generative evals?
Two structural fixes, best used together:
- Held-out evals keep part of the case set — or all of it — out of circulation. Every AgentGrading pack ships with roughly a fifth of its cases sealed and never delivered, so no buyer ever holds the complete answer key and the pack cannot be republished as a public target to optimize against. The strongest form of the same idea is to certify against cases the submitter has never seen: the agent goes to the eval rather than the eval going to the agent. What you never ship cannot be contaminated.
- Generative evals sell a generator, not a fixed set: each run draws fresh instances from a large or unbounded space. Memorizing last run's cases doesn't help, because the next run is different. This works best for structured tasks; validate generated cases and watch coverage, since synthetic data clusters around common patterns.
How often should you refresh an eval?
Treat evaluation as perishable. Refresh when contamination signals appear, and whenever the ground shifts underneath it — new models, new attack patterns, or changed standards. The healthy model is freshness-as-maintenance: rotate cases continuously, and once a set has clearly decayed, retire it and publish the expired version openly, where a leaked case becomes free reference content instead of a loss.
Contamination is one of two ways an eval dies. The other is the frontier of agent capability moving past what the eval can still tell apart — related, but measured differently, and covered in why AI agent evals go stale.
Related
- Why AI agent evals go stale — the other decay mechanism: capability moving past what the eval can discriminate.
- How to detect a gameable eval — contamination's structural sibling, where the rubric itself can be exploited.
- What is a reference-panel harness — how discriminating power gets measured in the first place.
- The medical RAG groundedness pack — a verified pack that ships with a sealed, undelivered slice of its cases.
- Benchmarks — where every pack's discriminating power is published.
FAQ
An eval is contaminated when a model scores well because it has already seen the test cases — in training data, blogs, or CI logs — rather than because it is genuinely good. The eval then measures memory, not skill.
Run a perturbation test: if a model passes the original case but fails a trivially reworded variant, it memorized the answer. Canary strings and sudden effortless frontier-model scores are further signals.
Once cases are public — in a repo, a blog, or CI logs — they flow into training data and get memorized. The more an eval is shared, the faster it loses discriminating power.
A held-out eval keeps some or all of its cases out of circulation, so the party being measured never holds the complete answer key. Nothing they were never given can be memorized or optimized against.
Refresh when contamination signals appear or when models, attacks, or standards shift. Treat freshness as ongoing maintenance, not a one-time build — a stale eval quietly stops discriminating.
Partly. A generator that draws fresh instances each run resists memorization of specific cases, but validate generated cases and watch for coverage gaps, since synthetic data clusters around common patterns.