Latency multiplies in a loop
Latency multiplies in a loop means an agent’s user-visible wait time grows with the sequential chain of model calls, tool calls, database queries, retries, and pauses it must complete. When each step depends on the previous result, the wall-clock time is roughly the sum of those waits, not the latency of one call.
This matters because agent workflows are often written as loops: think, call a tool, observe, think again, call another tool, then answer. A single slow operation may be acceptable, but a chain of dependent operations turns small waits into a long user pause. The painful part is not just total compute. It is that the next useful action cannot begin until the current one returns.
The concrete way to reason about it is a critical path diagram. List every model call, retrieval, tool invocation, validation, retry, and wait. For each item, note what it needs before it can start. The visible latency is the longest dependency chain that must finish before the response can be returned. Independent branches can overlap; dependent reasoning turns, where each consumes the previous output, sit in series.
The common mistake is multiplying an average step time and stopping there. Real systems are dominated by tail latency: with many calls, the chance that at least one is unusually slow becomes important. Retries also add elapsed time before they add progress, because a timeout has already consumed a wait. Optimising only the mean can make dashboards look better while users still experience slow completions.
Engineers meet this when designing agent loops, MCP tool plans, retrieval pipelines, and production latency budgets. The useful question is not only "how fast is this call?" but "is this call on the critical path?" Practical fixes include batching, prompt caching, narrower tool calls, speculative work, early exits, fewer reasoning turns, and parallelising independent retrievals or checks. What helps most depends on the dependency graph.
Common questions
- Is this just saying agents are slower because they do more work?
- Not exactly. Extra work matters, but the key issue is dependency. If work can run in parallel, it may add cost without adding much wall-clock time. Latency multiplies when each operation must wait for the previous one, so the user experiences the accumulated delay along the critical path.
- Why are average latencies misleading for multi-step agents?
- Averages hide slow cases. In a loop with many calls, a single tail event can dominate the whole response time. If retries are involved, the system may first wait for a timeout and then perform another attempt. For user experience, high-percentile latency is often a better planning number than the mean.
- How do I decide what to parallelise?
- Sketch the workflow as dependencies, then look for operations that do not need each other’s outputs. Retrievals from separate sources, independent validations, scoring passes, and prefetches are often candidates. Steps that require the previous model response or tool result usually remain serial unless you redesign the workflow.