12.04 · Concept
Knowing it broke before a user says so
Alert on the agent-specific signals: loop length, tool error rate, cost per run.
Agent failures often show up first as abnormal run behaviour: long reasoning loops, repeated tool calls, rising tool failures, retries, timeouts, and increasing cost per successful run. Monitoring the agent’s internal steps lets teams catch silent breakage before it becomes a bad answer, a slow response, or an expensive production incident.
What this lesson answers
- how to alert on agent loop length
- which agent metrics catch failures early
- how to monitor tool error rate in agents
Notes
Traditional service alerts watch things like latency, error rate, CPU, and saturation. Agentic systems need those too, but they also need signals about the reasoning loop itself. An agent can return HTTP 200 while silently wasting tokens, retrying the same tool, choosing the wrong tool, or getting stuck in a plan-act-observe cycle that never converges. The user-visible failure may arrive minutes later as a bad answer or an expensive run, so the alert has to fire on the process, not just the final response.
A useful mental model is to treat each agent run like a distributed trace.
Common questions
- Why are normal service alerts not enough for agents?
- An agent can complete the API request successfully while doing the wrong work internally. It may retry tools, burn tokens, loop through planning steps, or hide a tool outage behind a vague fallback answer. Latency and process errors still matter, but they do not expose whether the agent’s reasoning path is healthy.
- What should be captured for each agent run?
- Treat a run like a trace. Capture model calls, tool calls, selected tool names, tool outcomes, retry count, loop length, timeout behaviour, token use, wall-clock duration, final answer status, and estimated cost. That gives enough context to spot loops, brittle dependencies, failing tools, and cost drift.
- Which agent alerts should page an engineer?
- Page on conditions that are likely to affect users or create material cost risk: runaway loops, many runs exceeding a step budget, sharp tool failure increases, timeout spikes, or cost per successful run moving far above the normal baseline. Lower-severity anomalies are better routed to tickets or chat alerts.
Short definition: what is Knowing it broke before a user says so?