Budgets and circuit breakers
Budgets and circuit breakers are run-scoped guardrails that prevent an agent or LLM workflow from exceeding allowed cost, time, calls, tokens, retries, or tool use. The budget records what remains; the circuit breaker checks it before expensive actions and stops, downgrades, or falls back before runaway work continues.
LLM agents can spend unpredictably because they do not just make one request. They may call a model repeatedly, expand prompts, retry failures, invoke tools, or get trapped in planning loops. Looking at logs or bills after the run is not protection; by then the damage is already done. The point of a budget is to make the limit active during execution, so a single bad run cannot consume unbounded money or time.
The mechanism is usually simple: create a budget object at the start of a run and pass it through the workflow. Before each costly step, the code estimates or accounts for the planned action, such as a model call, token allowance, retry, tool invocation, or elapsed time. If enough allowance remains, the action proceeds and the budget is decremented. If not, the circuit breaker returns a controlled error, a partial answer, a cheaper path, or a request for confirmation.
The trade-off is that budgets can stop useful work as well as wasteful work. Tight limits improve predictability but may produce incomplete answers, more fallbacks, or extra user interactions. Loose limits reduce false stops but leave more exposure to loops and retries. The honest answer is usually workload-specific: limits should reflect task value, model cost, expected prompt size, tool latency, and how much failure the product can tolerate.
Engineers meet budgets and circuit breakers in agent runtimes, middleware around model calls, orchestration code, and production service boundaries. Common controls include call limits, token ceilings, retry caps, wall-clock deadlines, and estimated spend limits. A frequent misunderstanding is treating a timeout as sufficient. Timeouts cap latency, but a fast loop can still burn through many calls unless calls, tokens, or cost are checked too.
Common questions
- How is a budget different from monitoring?
- Monitoring observes what happened; a budget changes what is allowed to happen. Traces, logs, and billing reports are useful for tuning limits, but they are retrospective. A run-scoped budget is checked before expensive actions, so the workflow can stop, downgrade, or ask for confirmation while there is still something left to protect.
- What should a circuit breaker do when the budget is exhausted?
- It should fail deliberately, not crash ambiguously or continue silently. Good behaviours include returning partial results, explaining that the limit was reached, switching to a cheaper model if allowed, skipping optional work, or asking the user to approve more work. The right fallback depends on whether completeness, latency, or cost matters most.
- Is a timeout enough to control LLM agent cost?
- No. A timeout limits how long a run can continue, which helps with latency and stuck operations. It does not necessarily limit how many model calls, retries, tokens, or tool invocations happen inside that time. Cost-sensitive systems usually need time limits plus explicit caps on calls, tokens, retries, or estimated spend.