How to Measure Tool-Calling Accuracy in AI Agents

Tool calling is where capable agents quietly break. An agent can reason well and still pick the wrong function, pass a malformed argument, or plow on after a tool returns an error. Because these failures look plausible, a single "did the task succeed" score hides them. Tool call accuracy — also written tool calling accuracy — is the measurement that surfaces them, and measuring it properly means scoring each part of the call separately instead of collapsing a whole trace into one number.

Quick answer: Tool call accuracy measures whether an agent picks the right tool, passes correct arguments, sequences calls correctly, and recovers from tool errors. Score it deterministically where possible — exact argument match, schema validation, type checks — and reserve an LLM judge for free-form fields. Measure selection, arguments, sequencing, and error-handling separately, because one pass/fail hides where it breaks.

What is tool call accuracy?

Tool call accuracy is not one number. It is a set of dimensions, each of which can fail independently:

• Tool selection — did the agent choose the right tool for the request (and avoid calling one it didn't need)?
• Argument correctness — are the arguments valid against the tool's schema, and semantically right for the request?
• Call sequencing — for multi-step tasks, did it call tools in a workable order and pass results forward correctly?
• Error recovery — when a tool returns an error or empty result, does the agent retry, adjust, or abstain rather than fabricate?
• Clarification vs. guessing — on an under-specified request (a booking with no date, a lookup with an ambiguous name), does the agent ask instead of inventing an argument? This is the tool-calling analog of abstention in a RAG agent, and it's the check homegrown evals most often skip. Confidently guessing a missing parameter is the failure; asking is the correct behavior and should score full marks.

Collapsing these into one pass/fail is the most common measurement mistake. Each dimension fails for a different reason and is fixed in a different place, so a single number tells you that the agent broke without telling you where — which is the only part you can act on.

Which failure modes actually matter?

The dangerous failures are the ones that look fine. A call to the correct tool with a subtly wrong argument. A correct first call that the agent never recovers from once the tool errors. An unnecessary call that burns latency and cost. Happy-path testing — where the tool always returns clean data and the request is well-formed — creates false confidence, because production is never that clean. Weight your cases toward the messy paths.

A worked example: one trace, four dimensions

Abstract dimensions are easy to nod along to and hard to implement. Here is a single request scored end to end, with the actual pass/fail logic for each dimension.

Request: "Refund my last order and email me the confirmation." Tools available: find_order(customer_id), refund_order(order_id, amount_cents, reason), send_email(to, template, vars).

The agent's trace:

1  find_order(customer_id="C-8871")
   → {"order_id": "O-4412", "total_cents": 4250, "placed": "2026-05-02"}

2  refund_order(order_id="O-4412", amount_cents=4250, reason="customer_request")
   → {"error": "refund_window_expired"}

3  send_email(to="c8871@example.com", template="refund_confirmation",
             vars={"amount": "€42.50"})
   → {"sent": true}

Scored dimension by dimension:

This is the point of splitting the dimensions. A single "did the task succeed" check reports one bit — failed — and sends you looking at the wrong thing, because two of the four dimensions were perfect. The agent picked the right tools and passed flawless arguments; it simply did not read the error. That is a prompt-and-control-flow fix, not a tool-schema fix, and only per-dimension scoring tells you so.

Note also what the trace would score if refund_order had succeeded: four passes, and a clean run. The dimensions that break are the ones you only see when a tool fails — which is why your case set has to make tools fail on purpose.

Deterministic checks vs. LLM judges — when to use which?

Prefer determinism. It is faster, cheaper, and reproducible:

• Tool name → exact match against the expected tool.
• Structured arguments → schema validation, type checks, and exact or normalized match.
• Call count → assert the agent didn't make extra or missing calls.

Normalize before you compare, or determinism turns into false negatives. "NYC" and "New York City", or an ISO date and a spelled-out one, are the same argument in different surface forms — a naive exact-match fails the agent for choosing a different valid spelling of the right value. Canonicalize dates and known aliases first, then compare, so the check fails a genuine value error and not a harmless phrasing difference.

Taken dimension by dimension, most of the work is deterministic and only two edges genuinely need a judge:

The pattern holds across the table: determinism decides whether the behaviour was correct, and a judge only ever grades how well something free-form was expressed. If a judge is deciding correctness on a closed-set field, you have bought variance for no signal.

Only reach for an LLM-as-judge where a field is genuinely free-form and meaning matters more than string equality — for example, a natural-language query argument where several phrasings are equally correct. Even then, validate the judge periodically against human labels, because judge bias and drift quietly corrupt your numbers: a judge that rewards verbosity or favours whichever candidate it sees first will hand you a stable, precise, wrong score.

The split, by argument type:

The rule underneath the table: use a judge only where the set of correct answers is open. Every closed-set field scored by a judge is cost and variance you bought for no additional signal.

What does argument-level scoring look like in practice?

"The agent called the right tool" is not a result you can act on. Score each argument separately, then decide how the parts roll up. Take a booking request — "Book a flight from Amsterdam to New York on 3 March for two people" — against book_flight(origin, destination, date, passengers):

Two things fall out of this table. First, the near-miss is invisible to any pass/fail that only checks the tool name and schema validity — it is the failure mode most likely to reach production, and it is only caught by comparing argument values. Second, the last row is scored on the opposite polarity: the agent that confidently fills in a missing date scores worse than the one that asks.

How the per-argument scores roll up is a deliberate choice. All-or-nothing (every argument must match) is the right default for calls with side effects — booking, payment, writes — because a call that is 75% correct still books the wrong flight. Partial credit is more useful for tracking progress on read-only calls, where a 3/4 tells you the agent is close. Report both if you can: the strict rate gates the release, the partial rate tells you which argument is costing you.

Tool calling accuracy vs. task success rate

These get used interchangeably and they are not the same measurement. Task success rate is end-to-end: did the user get what they asked for, one bit per task. Tool calling accuracy is per-call: was each step of the path correct. One is the outcome, the other is the mechanism.

They come apart in both directions, which is why you need both:

• High task success, poor tool accuracy. The agent makes redundant calls, guesses an argument and gets lucky, or recovers by brute force. The task completes, while cost and latency quietly climb and the near-miss arguments wait for an edge case to bite.
• High tool accuracy, failed task. Every call is exactly what you specified, and the task still fails — because the tool itself returned bad data, or the golden set encoded a path that no longer achieves the goal. This one is valuable: it points at the tooling or the case set rather than the agent.

The practical rule: report task success rate to the people deciding whether to ship, and tool call accuracy to the people fixing it. Gate CI on the per-dimension numbers, because those are the ones that localise a regression. If you only track task success, a regression tells you something broke last week and leaves you to find it by hand.

How do you build a golden set for tool calls?

Combine two sources. Mine real production traces for authentic cases and label them, then add synthetic edge cases to cover failure paths you haven't seen yet — ambiguous requests, missing parameters, tools that error, and near-miss arguments. Synthetic generation is fast but clusters around common patterns, so always review generated cases before they enter your official set.

How big should the set be? Big enough for the risk it carries. Volume is only meaningful when scored against a minimum that rises with the deployment's risk tier, which is why a high-risk pack with fifteen cases scores worse than a low-risk one with the same fifteen. Coverage of the failure paths beats raw count. And keep the set out of places it can leak: a tool-calling eval that ends up in public CI logs loses its value once agents are trained on it — see how to tell if your eval is contaminated.

How do you know your tool-calling eval actually works?

Before you trust an eval's verdicts on an agent you don't know, run it against agents whose quality you already know. This is mutation testing applied to the eval itself:

• A known-good agent (right tool, right arguments, asks when unsure) should score high.
• A known-broken agent (loose argument discipline, occasional guessing) should score visibly lower.
• A known-sabotaged agent (deliberately plausible-but-wrong calls) should score near the floor.

If the eval can't separate those three, it isn't measuring tool-calling — it's measuring something merely correlated with it, like response length. See the tool-calling correctness benchmark for this discriminating-power check run on a real reference panel, and what a reference-panel harness is for the method in depth.

How do you catch regressions in CI?

Treat tool-calling evals like unit tests. Every pull request that touches a prompt, a model version, or a tool schema triggers an eval run against the golden set, and a run that regresses past your threshold does not merge. Track the dimensions separately over time — a drop in error-recovery rate tells you something very different from a drop in tool selection.

CI is only half the picture, though. A golden set answers a closed question — does the agent still handle the cases we thought of? — and it cannot see the inputs you never imagined. Once the agent ships, pair the gate with a sensor: monitoring tool calls in production tracks selection distribution, argument validity, and unrecovered errors on live traffic, and feeds the failures it finds back into this golden set as new cases. It's also a different question from everything above: this section is about whether the agent still gets the decision right, not whether it holds up against a slow or flaky tool — see tool-calling accuracy vs reliability for the execution-time failures a clean golden set never exercises.

Related

• Tool-calling accuracy vs reliability — why passing this eval is not the same as holding up against timeouts, retries, and malformed responses.
• How to monitor tool calls in production — the production half of this pair, once the eval has gated the release.
• LLM judge bias — validating the judge you use for free-form arguments.
• What is a reference-panel harness — the discriminating-power method behind the "does the eval work" check above.
• How to tell if your AI eval is contaminated — why leaked cases quietly destroy these numbers.
• Why AI agent evals go stale — keeping this eval trustworthy over time.
• The tool-calling correctness pack — a verified, ready-to-run suite.
• Tool-calling correctness benchmark — discriminating power measured on a real reference panel.
• Capability packs — the full category.

← Back to guides