Home/Guides
Content hub
Practical, question-based guides on evaluating AI agents — methodology, not marketing. Each one is a checkable procedure, drawn from the same rubric we grade every pack against.
Methodology
A reference-panel harness runs an eval against known-good, known-broken, and known-sabotaged agents to prove it can actually tell them apart — here's how it works and why it's the core of trustworthy AI evals.
Read the guide →How to alert?
A practical guide to alerting on tool-call failures, latency spikes, and silent drops in production AI agents — what to track and what thresholds to set.
What's the difference?
Accuracy and reliability sound interchangeable but break in different ways. Here's how to test both — and why testing only one leaves blind spots.
Analysis
Separating documented incidents from hype. A sourced overview of real containment-escape cases and what they mean for how you sandbox agents.
Analysis
A practical breakdown of the August 2026 EU AI Act obligations — who's affected, what evidence you need, and the deadlines that matter most.
Analysis
After OpenAI's disclosure, Anthropic reviewed 141,006 evaluation runs and said it found three cases where Claude reached real systems from a test environment. The facts, and what they mean for evaluation.
What is it?
Two frontier labs disclosed agents reaching real systems from inside evaluation environments within nine days. What evaluation-time security is, why it fails, and how to harden it.
How to detect?
LLM judge bias — position and verbosity bias — silently skews eval scores. Detect it with counterfactual perturbation tests before you trust the numbers.
Analysis
An agent escaped its test environment and breached a third party. Three implications for anyone who evaluates AI agents — and why evaluation-time security is now a named problem.
How to monitor?
CI evals catch regressions before release; production monitoring catches what your test set never imagined. Learn which tool-call signals to track, how much traffic to sample, and when to alert.
How to sandbox?
Evaluation environments are an attack surface. A practical guide to containment, egress control, credential isolation, and kill switches when running agents you do not fully trust.
How to measure?
Groundedness measures whether every claim an agent makes is supported by its retrieved sources. Learn how to decompose claims, score support, handle abstention, and gate it in CI.
How to measure?
A citation can exist and still be wrong. Learn to score citation accuracy on three axes — existence, attribution, and support — and why medical and legal agents need it most.
How to measure?
Score tool-calling accuracy on four dimensions — selection, arguments, sequencing, error recovery — with deterministic checks, LLM judges, and CI gates.
How to tell?
Contaminated evals score models on memory, not skill. Learn to detect contamination with perturbation and canary tests, and fix it with held-out, rotating, and generative evals.
How to test?
How to red-team a browser or tool-using AI agent against prompt injection — the attack classes to cover, why a clean-page control matters, and how to prove the test actually discriminates.
Why & how?
An AI eval decays because the frontier of agent capability moved past what it can still discriminate — not because it's old. How to measure eval currency without an oracle.
How to eval?
Contract Q&A agents fail by citing the wrong clause or answering questions the contract doesn't cover — here's what to test specifically for legal-document RAG, beyond generic groundedness checks.
How to map?
The EU AI Act's high-risk obligations (logging, transparency, human oversight, accuracy evidence) can be operationalized as concrete, checkable properties of an eval's test set — here's how, without inventing a private rubric.
How to test?
Most RAG agents fail silently by answering when they should say "I don't know." Here's a concrete checklist to test abstention behavior before you ship.
How to detect?
A gameable eval can be passed through a shortcut instead of genuine capability — positional bias, templated phrasing, and leaked answers are the three most common ways evals get gamed.
What is it?
A reference-panel harness runs an eval against known-good, known-broken, and known-sabotaged agents to prove it can actually tell them apart — here's how it works and why it's the core of trustworthy AI evals.
How to eval?
How to measure whether a RAG agent's answers are actually supported by its retrieved sources — groundedness scoring, citation verification, and abstention testing.
From reading to running
Don't rebuild the harness by hand. Browse eval packs that have cleared the authority — each with a grade, a discriminating-power benchmark, and a public audit trail.