EU AI Act — high-risk conformance
Evidence-based checklist for high-risk obligations: logging, human oversight, transparency, data governance, and conformity assessment. Anchored to the Act, not to opinion.
The method this pack implements is written up in How to map an eval to the EU AI Act.
Verification report
Six axes, one grade
Every pack is measured on the same six axes — including whether it stays discriminating under perturbed inputs, and whether it's still current against the moving capability frontier. No oracle, no self-reporting — reproducible measurement only.
Structural validity
Reproducible, seed-stable scoring with no leakage from the golden set into the rubric — provable without knowing the ‘best’ eval.
No data leakage 1.00 · Ungameable 1.00 · Deterministic 1.00
Discriminating power
Cleanly separates strong from weak agents on the reference panel; the sabotaged reference agent should fail.
Standard coverage
Anchored to EU AI Act — Art. 53 / high-risk, ISO/IEC 42001.
Test thoroughness
Test volume weighed against the minimum expected for a 'high risk' pack — not a flat headcount.
8 obligations
Robustness
Holds its discriminating power when the test items are reordered or padded with distractors — so the grade tracks agent behavior, not the exact surface form of the frozen test set.
Robustness across the input space
Currency
How current the pack is against the moving frontier of agent capability. An eval today's best agents have saturated no longer discriminates, however well-built — this decays over time and resets on revalidation. Not calendar age.
Fresh — validated against the current frontier cohort
What this grade does and doesn’t prove — published on purpose, not hidden in fine print.
- Reference agents are controlled behavioral archetypes (good/broken/sabotaged), not live production models — validate against your own agent's real outputs before treating this grade as a guarantee for your specific deployment.
- Does not yet cover: Post-market monitoring plan, Serious-incident reporting to authorities — see the standard-coverage checklist above.
Reference panel
How the panel performed
Reference agents — from strong to deliberately sabotaged — ran the full pack. A trustworthy eval must rank them correctly.
A good pack scores the known-good agent high and the sabotaged one near zero. That gap is the evidence the meter works.
Output
What you get when you run it
Pytest-style pass/fail output per test, with each metric's score printed to console (and exportable via `--output`).
- pass or fail per test case against each configured metric (e.g. Faithfulness, AnswerRelevancy)
- a 0-1 score per metric per test case
- the standard pytest summary (counts, failures, timing) your CI already parses
`deepeval test run` is a pytest wrapper — it fails the build on any metric threshold miss exactly like a normal failing test would.
The known-good reference agent on this pack scores 1.00 — if your agent's aggregate score lands well below that, treat it as a signal to investigate before shipping, not a false alarm.
Real content from this pack's own test set, run through the known-good reference agent:
Obligation/status/evidence are real content from this pack's own checklist, shown against the known-compliant reference submission — not a fabricated illustration.
This pack's score is already obligation-by-obligation — read each row as a line item in your own gap analysis, not just a total.
| Category | What it measures | A low score here means |
|---|---|---|
| Risk-management system across the AI system's lifecycle | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Technical documentation | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Logging capable of enabling traceability | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Transparency and instructions for use | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Human-oversight measures | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Accuracy, robustness and cybersecurity evidence | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Data governance and quality management | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
| Conformity assessment before market placement | Whether your submission provides real, checkable evidence for this specific obligation — not just a yes/no claim. | A 'no' or an unproven 'yes' here is a concrete gap in your own conformance file for this obligation. |
Integration
One line in CI
Runs locally or in your pipeline. Fail the build the same way a failing test would.
Anchored standards
EU AI Act obligation coverage
Checkable high-risk obligations this pack's own test cases exercise:
- Risk-management system across the AI system's lifecycle
- Technical documentation
- Logging capable of enabling traceability
- Transparency and instructions for use
- Human-oversight measures
- Accuracy, robustness and cybersecurity evidence
- Data governance and quality management
- Conformity assessment before market placement
- Flags a non-compliant submission (catches the non-compliant reference)
- Post-market monitoring plan — not covered
- Serious-incident reporting to authorities — not covered
Not legal advice — see the EU AI Act standard page for the source text these checks are drawn from.
Certification
Ship with a certified meter
Add EU AI Act — high-risk conformance to your pipeline today — a signed grade, reproducible measurement, zero oracle.
Get this pack — €99