How to Set Up Tool-Call Monitoring Alerts for Production AI Agents
Monitoring an AI agent's tool calls in production is different from standard API monitoring. The agent decides when and how to call a tool, so failures can be silent — the agent simply doesn't call anything, or calls the wrong thing, and nothing throws an exception. Here's what to actually alert on.
Quick answer: Standard API alerts watch error rate, which misses an agent's most consequential failures — it doesn't throw when it silently stops calling a tool, calls the wrong one, or abandons a task after an error. Alert instead on call-volume drops per tool, argument-shape drift, retry loops, per-tool latency percentiles, and tasks that end without a defined done/failed state.
If you haven't set up baseline tool-call logging yet, start with our guide on how to monitor tool calls in production for the logging schema this builds on.
What to alert on (and why standard API alerts miss it)
1. Call-rate drop, not just error rate
A standard API monitor alerts when error rate spikes. For agents, the dangerous signal is often the opposite: a sudden drop in tool-call volume for a given tool, which usually means the agent stopped deciding to call it — often because a prompt change, model update, or upstream schema change broke the decision path. Alert on a percentage drop in call volume per tool, compared to a rolling baseline, not just on error codes.
2. Argument-shape drift
Log the schema of arguments passed to each tool, not just whether the call succeeded. Alert
when the distribution of argument shapes changes — e.g., a date field that's usually
ISO-8601 suddenly arriving as a natural-language string. This catches accuracy regressions
that a pure success/failure check won't.
3. Retry-loop detection
Set a threshold (e.g., more than 3 calls to the same tool with near-identical arguments within a single task) and alert on it. This is the signature of an agent stuck retrying without resolving the underlying issue — a reliability failure mode that looks "healthy" in aggregate success-rate dashboards because the call eventually succeeds. See tool-calling accuracy vs reliability for why this failure mode is invisible to a static accuracy eval in the first place.
4. Latency percentiles per tool, not global average
A global p50 latency hides the tool that's degrading. Track p95/p99 latency per individual tool. Agents often chain multiple tool calls per task, so one slow tool compounds across every task that uses it — alert per-tool, and set the threshold relative to that tool's own baseline, not a fixed global number.
5. Silent task abandonment
The hardest failure to catch: the agent receives a tool error, doesn't retry, and simply ends the task without completing it — no exception, no explicit failure state. Instrument task completion explicitly (a task either reaches a defined "done" state or an explicit "failed" state) and alert on tasks that end in neither.
A minimal alerting setup
| Signal | Threshold to start with | Why |
|---|---|---|
| Call volume drop (per tool) | >30% below 7-day rolling avg | Catches broken decision paths |
| Argument schema drift | Any new shape not in baseline set | Catches accuracy regressions |
| Retry loop | ≥3 near-identical calls per task | Catches reliability failures |
| Per-tool p95 latency | >2x that tool's 7-day baseline | Catches degrading dependencies |
| Undefined task end-state | Any occurrence | Catches silent abandonment |
Start with wide thresholds and tighten them once you have a week of production baseline data — alerting too aggressively on day one just trains the team to ignore the channel.
The takeaway
Tool-call monitoring isn't just uptime monitoring with extra steps. The agent's decision to call (or not call) a tool is itself a signal worth tracking, and the failures that matter most — silent drops, retry loops, argument drift — don't trip a standard error-rate alert. Instrument for agent behavior, not just endpoint health.
Related
- How to monitor tool calls in production — the logging and sampling foundation these alerts sit on top of.
- Tool-calling accuracy vs reliability — why retry loops and silent drops don't show up in a pre-release eval.
- How to measure tool-calling accuracy — the CI gate this alerting layer complements.
- The tool-calling correctness pack — a verified suite for the pre-release gate.
- Capability packs — the full category.
FAQ
A standard API monitor alerts on error rate — a thrown exception, a non-2xx response. An agent's most dangerous failures don't throw anything: it simply decides not to call a tool, or calls the wrong one, and the request that does go out still returns 200. The failure is in the agent's decision layer, which sits above the layer standard monitoring watches.
Five signals cover most of the risk: a drop in call volume per tool (the decision path broke), argument-shape drift (an accuracy regression), retry loops (a reliability failure hiding inside a healthy success rate), per-tool latency percentiles (a global average hides the one degrading tool), and undefined task end-states (silent abandonment).
It's a sudden decline in how often the agent calls a given tool, compared to its rolling baseline. Unlike an error-rate spike, a volume drop usually means the agent stopped deciding to call the tool at all — often from a prompt change, model update, or upstream schema change that broke the decision path silently, with no error anywhere in the logs.
It's a change in the distribution of argument values or formats passed to a tool — for example a date field that's normally ISO-8601 suddenly arriving as a natural-language string. Tracking only success/failure misses this; tracking the shape of the arguments catches accuracy regressions before they cause a downstream error.
Set a threshold — for example, three or more calls to the same tool with near-identical arguments inside a single task — and alert when it's crossed. This failure mode is easy to miss because the call often eventually succeeds, so it looks healthy in an aggregate success-rate dashboard even though the agent is stuck retrying instead of resolving the underlying issue.
It's when an agent receives a tool error, doesn't retry, and simply ends the task without completing it — no exception, no explicit failure state, nothing for a standard monitor to catch. Catching it requires instrumenting task completion explicitly, so every task reaches either a defined 'done' state or an explicit 'failed' state, and alerting on any task that ends in neither.
Start wide and tighten with data: a call-volume drop of more than 30% below the 7-day rolling average per tool, any new argument shape outside the observed baseline set, three or more near-identical retries per task, per-tool p95 latency more than 2x that tool's own 7-day baseline, and any occurrence of a task ending outside a defined done/failed state. Loosen or tighten each after a week of production baseline data.