API-docs RAG — groundedness & abstention
Checks that a developer-docs assistant answers from the actual API reference cited — rate limits, status codes, auth — and abstains on environment-specific questions it can't answer from the docs.
Verification report
Six axes, one grade
Every pack is measured on the same six axes — including whether it stays discriminating under perturbed inputs, and whether it's still current against the moving capability frontier. No oracle, no self-reporting — reproducible measurement only.
Structural validity
Reproducible, seed-stable scoring with no leakage from the golden set into the rubric — provable without knowing the ‘best’ eval.
No data leakage 1.00 · Ungameable 0.90 · Deterministic 1.00
Discriminating power
Cleanly separates strong from weak agents on the reference panel; the sabotaged reference agent should fail.
Standard coverage
Capability pack — measured for internal consistency, not an external standard.
Test thoroughness
Test volume weighed against the minimum expected for a 'low risk' pack — not a flat headcount.
15 test cases
Robustness
Holds its discriminating power when the test items are reordered or padded with distractors — so the grade tracks agent behavior, not the exact surface form of the frozen test set.
Robustness across the input space
Currency
How current the pack is against the moving frontier of agent capability. An eval today's best agents have saturated no longer discriminates, however well-built — this decays over time and resets on revalidation. Not calendar age.
Aging — one cohort behind; due for revalidation
What this grade does and doesn’t prove — published on purpose, not hidden in fine print.
- Reference agents are controlled behavioral archetypes (good/broken/sabotaged), not live production models — validate against your own agent's real outputs before treating this grade as a guarantee for your specific deployment.
- The test set covers 15 cases — enough to establish discriminating power, not exhaustive coverage of every question shape in this domain. Treat a pass as evidence the pack works, not proof your exact corpus is fully covered.
- Structural-validity score is 0.90, not a perfect 1.00 — some positional or phrasing regularity remains in the test set. Treat this as strong rather than absolute protection against a shortcut-taking agent.
- Discriminating power is 0.94 — the pack ranks known-good above known-sabotaged, but the margin isn't maximal. Borderline agents near the pass threshold deserve a closer manual look, not just the number.
- Currency is 0.66 — this pack was last validated against an earlier frontier cohort. Revalidate it against the current reference panel before treating the grade as a fresh signal; capability that discriminated agents then may be saturated now (see the currency axis).
Reference panel
How the panel performed
Reference agents — from strong to deliberately sabotaged — ran the full pack. A trustworthy eval must rank them correctly.
A good pack scores the known-good agent high and the sabotaged one near zero. That gap is the evidence the meter works.
Output
What you get when you run it
A per-test-case score table (exportable to JSON/CSV) plus an aggregate mean per metric.
- question, retrieved_contexts, and your agent's answer for every test case
- a faithfulness score (0-1) and an answer_correctness score (0-1) per test case
- an aggregate mean per metric across the whole test set
Assert the aggregate faithfulness/answer_correctness meets your threshold in your CI script; a non-zero exit on failure blocks the deploy the same way a failing unit test would.
The known-good reference agent on this pack scores 1.00 — if your agent's aggregate score lands well below that, treat it as a signal to investigate before shipping, not a false alarm.
A real sample of the scorecard this pack produces — actual numbers from the known-good and known-sloppy reference agents, not a mockup.
| Category | Known-good agent | Known-sloppy agent |
|---|---|---|
| Groundedness (answerable) | 1.00 | 0.70 |
| Abstention (unanswerable) | 1.00 | 0.40 |
0-1 score, per category — this is the shape of your own report once you run the pack.
Real content from this pack's own test set, run through the known-good reference agent:
Question/sources/answer are real content from this pack's own test set, run through the known-good reference agent — not a fabricated illustration. Frameworks like ragas additionally compute judge-model scores (e.g. faithfulness, answer correctness) from your actual agent's output when you run the pack yourself.
Your aggregate score breaks into two things worth checking separately, not just one number.
| Category | What it measures | A low score here means |
|---|---|---|
| Groundedness (answerable questions) | Whether the answer contains the right facts and cites the source that actually supports them. | The agent has the right evidence but gets facts wrong or cites the wrong source — a factual-accuracy problem. |
| Abstention (unanswerable questions) | Whether the agent says it doesn't know when the sources don't support an answer, instead of guessing. | The agent fabricates answers when it shouldn't — worse than a groundedness miss, since it looks confident while being wrong. |
Integration
One line in CI
Runs locally or in your pipeline. Fail the build the same way a failing test would.
Anchored standards
Capability pack — no external standard.
EU AI Act obligation coverage
A diagnostic signal, not a formal claim — checkable EU AI Act obligations this pack's test structure happens to exercise:
- Logging capable of enabling traceability
- Transparency and instructions for use
- Accuracy / robustness evidence
- Human-oversight measures (flags an untrustworthy system)
Not legal advice — see the EU AI Act standard page for the source text these checks are drawn from.
Certification
Ship with a certified meter
Add API-docs RAG — groundedness & abstention to your pipeline today — a signed grade, reproducible measurement, zero oracle.
Get this pack — €39