Tool-calling correctness
Verifies function/tool selection and argument correctness, and checks the agent asks for clarification instead of guessing when a request is ambiguous.
The method this pack implements is written up in How to measure tool-calling accuracy.
Verification report
Six axes, one grade
Every pack is measured on the same six axes — including whether it stays discriminating under perturbed inputs, and whether it's still current against the moving capability frontier. No oracle, no self-reporting — reproducible measurement only.
Structural validity
Reproducible, seed-stable scoring with no leakage from the golden set into the rubric — provable without knowing the ‘best’ eval.
No data leakage 1.00 · Ungameable 1.00 · Deterministic 1.00
Discriminating power
Cleanly separates strong from weak agents on the reference panel; the sabotaged reference agent should fail.
Standard coverage
Capability pack — measured for internal consistency, not an external standard.
Test thoroughness
Test volume weighed against the minimum expected for a 'medium risk' pack — not a flat headcount.
15 test cases
Robustness
Holds its discriminating power when the test items are reordered or padded with distractors — so the grade tracks agent behavior, not the exact surface form of the frozen test set.
Robustness across the input space
Currency
How current the pack is against the moving frontier of agent capability. An eval today's best agents have saturated no longer discriminates, however well-built — this decays over time and resets on revalidation. Not calendar age.
Aging — one cohort behind; due for revalidation
What this grade does and doesn’t prove — published on purpose, not hidden in fine print.
- Reference agents are controlled behavioral archetypes (good/broken/sabotaged), not live production models — validate against your own agent's real outputs before treating this grade as a guarantee for your specific deployment.
- The test set covers 15 cases — enough to establish discriminating power, not exhaustive coverage of every question shape in this domain. Treat a pass as evidence the pack works, not proof your exact corpus is fully covered.
- Only 15 test cases for a 'medium-risk' pack — 25 is the bar for full thoroughness credit here. Treat this grade as a starting signal, not exhaustive proof for a deployment at this risk level.
- Currency is 0.66 — this pack was last validated against an earlier frontier cohort. Revalidate it against the current reference panel before treating the grade as a fresh signal; capability that discriminated agents then may be saturated now (see the currency axis).
Reference panel
How the panel performed
Reference agents — from strong to deliberately sabotaged — ran the full pack. A trustworthy eval must rank them correctly.
A good pack scores the known-good agent high and the sabotaged one near zero. That gap is the evidence the meter works.
Output
What you get when you run it
A pass/fail result per test case (viewable via `promptfoo view`, exportable to JSON/CSV).
- pass or fail per test case, with the specific assertion that failed if any
- an aggregate pass-rate percentage across the test set
- the full prompt/response pair for every case, for manual review of failures
`promptfoo eval` exits non-zero when the pass-rate drops below the config's threshold, which fails the CI job automatically — no separate scripting needed.
The known-good reference agent on this pack scores 1.00 — if your agent's aggregate score lands well below that, treat it as a signal to investigate before shipping, not a false alarm.
A real sample of the scorecard this pack produces — actual numbers from the known-good and known-sloppy reference agents, not a mockup.
| Category | Known-good agent | Known-sloppy agent |
|---|---|---|
| Clear tool calls | 1.00 | 0.75 |
| Ambiguous requests (clarification) | 1.00 | 0.60 |
0-1 score, per category — this is the shape of your own report once you run the pack.
Real content from this pack's own test set, run through the known-good reference agent:
Request/tools/action are real content from this pack's own test set, run through the known-good reference agent — not a fabricated illustration.
Your aggregate score breaks into three things worth checking separately.
| Category | What it measures | A low score here means |
|---|---|---|
| Tool selection | Whether the agent picks the correct tool for the request, out of the distractor tools also on offer. | The agent is reaching for the wrong capability — often a sign its tool descriptions are ambiguous or overlapping. |
| Argument accuracy | Whether the values passed to the chosen tool match what the request actually specified. | The agent picks the right tool but fills it in wrong — check argument parsing/extraction, not tool routing. |
| Clarification-seeking | Whether the agent asks before acting when the request is missing information, instead of guessing. | The agent guesses instead of asking — a common but serious failure mode, since a wrong guess executes silently. |
Integration
One line in CI
Runs locally or in your pipeline. Fail the build the same way a failing test would.
Anchored standards
Capability pack — no external standard.
Certification
Ship with a certified meter
Add Tool-calling correctness to your pipeline today — a signed grade, reproducible measurement, zero oracle.
Get this pack — €49