Insurance policy RAG — groundedness & abstention
Checks that a policy/claims assistant answers from the exact clause it cites — deductibles, limits, exclusions — and abstains on claim-specific questions it has no evidence for, instead of inventing coverage.
Verification report
Six axes, one grade
Every pack is measured on the same six axes — including whether it stays discriminating under perturbed inputs, and whether it's still current against the moving capability frontier. No oracle, no self-reporting — reproducible measurement only.
Structural validity
Reproducible, seed-stable scoring with no leakage from the golden set into the rubric — provable without knowing the ‘best’ eval.
No data leakage 1.00 · Ungameable 0.90 · Deterministic 1.00
Discriminating power
Cleanly separates strong from weak agents on the reference panel; the sabotaged reference agent should fail.
Standard coverage
Anchored to EU AI Act — Art. 53 / high-risk.
Test thoroughness
Test volume weighed against the minimum expected for a 'high risk' pack — not a flat headcount.
15 test cases
Robustness
Holds its discriminating power when the test items are reordered or padded with distractors — so the grade tracks agent behavior, not the exact surface form of the frozen test set.
Robustness across the input space
Currency
How current the pack is against the moving frontier of agent capability. An eval today's best agents have saturated no longer discriminates, however well-built — this decays over time and resets on revalidation. Not calendar age.
Fresh — validated against the current frontier cohort
What this grade does and doesn’t prove — published on purpose, not hidden in fine print.
- Reference agents are controlled behavioral archetypes (good/broken/sabotaged), not live production models — validate against your own agent's real outputs before treating this grade as a guarantee for your specific deployment.
- The test set covers 15 cases — enough to establish discriminating power, not exhaustive coverage of every question shape in this domain. Treat a pass as evidence the pack works, not proof your exact corpus is fully covered.
- Only 15 test cases for a 'high-risk' pack — 50 is the bar for full thoroughness credit here. Treat this grade as a starting signal, not exhaustive proof for a deployment at this risk level.
- Structural-validity score is 0.90, not a perfect 1.00 — some positional or phrasing regularity remains in the test set. Treat this as strong rather than absolute protection against a shortcut-taking agent.
Reference panel
How the panel performed
Reference agents — from strong to deliberately sabotaged — ran the full pack. A trustworthy eval must rank them correctly.
A good pack scores the known-good agent high and the sabotaged one near zero. That gap is the evidence the meter works.
Output
What you get when you run it
A per-test-case score table (exportable to JSON/CSV) plus an aggregate mean per metric.
- question, retrieved_contexts, and your agent's answer for every test case
- a faithfulness score (0-1) and an answer_correctness score (0-1) per test case
- an aggregate mean per metric across the whole test set
Assert the aggregate faithfulness/answer_correctness meets your threshold in your CI script; a non-zero exit on failure blocks the deploy the same way a failing unit test would.
The known-good reference agent on this pack scores 1.00 — if your agent's aggregate score lands well below that, treat it as a signal to investigate before shipping, not a false alarm.
A real sample of the scorecard this pack produces — actual numbers from the known-good and known-sloppy reference agents, not a mockup.
| Category | Known-good agent | Known-sloppy agent |
|---|---|---|
| Groundedness (answerable) | 1.00 | 0.78 |
| Abstention (unanswerable) | 1.00 | 0.40 |
0-1 score, per category — this is the shape of your own report once you run the pack.
Real content from this pack's own test set, run through the known-good reference agent:
Question/sources/answer are real content from this pack's own test set, run through the known-good reference agent — not a fabricated illustration. Frameworks like ragas additionally compute judge-model scores (e.g. faithfulness, answer correctness) from your actual agent's output when you run the pack yourself.
Your aggregate score breaks into two things worth checking separately, not just one number.
| Category | What it measures | A low score here means |
|---|---|---|
| Groundedness (answerable questions) | Whether the answer contains the right facts and cites the source that actually supports them. | The agent has the right evidence but gets facts wrong or cites the wrong source — a factual-accuracy problem. |
| Abstention (unanswerable questions) | Whether the agent says it doesn't know when the sources don't support an answer, instead of guessing. | The agent fabricates answers when it shouldn't — worse than a groundedness miss, since it looks confident while being wrong. |
Integration
One line in CI
Runs locally or in your pipeline. Fail the build the same way a failing test would.
Anchored standards
EU AI Act obligation coverage
Checkable high-risk obligations this pack's own test cases exercise:
- Logging capable of enabling traceability
- Transparency and instructions for use
- Accuracy / robustness evidence
- Human-oversight measures (flags an untrustworthy system)
Not legal advice — see the EU AI Act standard page for the source text these checks are drawn from.
Certification
Ship with a certified meter
Add Insurance policy RAG — groundedness & abstention to your pipeline today — a signed grade, reproducible measurement, zero oracle.
Get this pack — €89