How to Monitor Tool Calls in Production (2026)

A tool-calling eval in CI answers a closed question: does the agent still handle the cases we thought of? Production asks an open one: what is it doing on the inputs we never imagined? Both matter, and they catch different failures. An agent that passes every CI case can still degrade in production when a tool schema changes, a model updates, or users start asking things your golden set never contained.

Quick answer: Tool call monitoring is the continuous observation of an agent's tool usage in production — which tools it picks, whether arguments validate, how often calls fail, and how those patterns drift. Capture cheap deterministic signals on all traffic, score a 5-10% sample more deeply, and alert on distribution shifts rather than only on hard errors.

How does monitoring differ from CI evaluation?

CI evaluation is a gate: a fixed golden set, a threshold, a merge that passes or blocks. Production monitoring is a sensor: live traffic, no ground truth, and the goal is noticing that something changed. The distinction matters because they fail in opposite ways. CI misses what is not in the set; monitoring sees everything but cannot tell you what "correct" was.

Run both. The tool-calling accuracy eval tells you the agent was right yesterday against known cases; monitoring tells you today's behaviour has drifted away from that baseline.

Which signals should you track?

Split them by cost. Some are nearly free and should run on all traffic:

• Argument validity rate — does each call validate against the tool's schema? Pure deterministic check.
• Tool error rate — how often does the tool itself return an error or empty result?
• Unrecovered error rate — how often does the agent continue as if a failed call succeeded? This is the most consequential signal, because it is where fabrication starts.
• Latency and cost per call — cheap, and a leading indicator of loops.

Others need scoring, so run them on a sample:

• Tool selection appropriateness — was this the right tool for the request?
• Unnecessary call rate — calls that added latency and cost without changing the outcome.
• Sequencing quality — for multi-step tasks, did results flow correctly between calls?

Track everything per tool, not just in aggregate. One badly-performing tool disappears inside a healthy average.

What does drift look like, and how do you spot it?

The most useful signal is not a threshold breach but a change in distribution. If a tool that handled 10% of calls last week is suddenly handling 35%, something moved — a prompt edit, a model update, a change in what users are asking. Distribution shifts usually appear before user-visible failures, which makes them your earliest warning.

Practical baseline set:

• Tool selection distribution, week over week.
• Argument validation failure rate, per tool.
• Unrecovered error rate — this one should be near-zero, so treat any sustained rise as a page-worthy alert.
• Retry rate, which quietly absorbs problems until it doesn't.

How much should you sample?

Capture the deterministic signals on 100% of traffic — they cost almost nothing. For the scored signals that need an LLM judge, sampling in the 5-10% range is a common starting point, weighted toward interesting traffic rather than uniformly: cases that errored, retried, took unusually long, or used rare tools. Uniform sampling spends most of your budget confirming that the happy path is still happy.

If you use a judge for the scored signals, validate it first. A judge with position or verbosity bias will produce a monitoring dashboard that is stable, precise, and wrong.

How does monitoring feed back into evaluation?

This is the part most teams underuse. Production is the highest-value source of new test cases, because it contains failures you could not have invented. The loop:

1. Flag interesting traces — errors, retries, unusual tool sequences, low scored samples.
2. Label them, keeping the surrounding context.
3. Promote the durable ones into your golden set so the failure cannot silently recur.
4. Rotate older cases out as they age, so the set keeps discriminating rather than going stale.

One caution: cases mined from production and then published — in a blog, a public repo, or verbose CI logs — lose their value quickly. That is contamination arriving through your own pipeline. Keep the golden set private if you want it to keep measuring anything.

Related

• How to measure tool-calling accuracy — the CI half of this pair.
• The tool-calling correctness pack — a verified suite for the gate.
• Tool-calling correctness benchmark — discriminating power on a real reference panel.
• LLM judge bias — validate the judge before you trust the dashboard.
• How to tell if your AI eval is contaminated — why mined cases leak their value.

← Back to guides