11.04 · Concept
Latency multiplies in a loop
Predict wall-clock for an N-step agent and find what to parallelise.
Sequential agent latency is governed by the critical path, not by one model call. Each dependent model call, tool call, query, retry and wait adds to user-visible time. Estimating wall-clock means mapping dependencies, using tail latency rather than averages, and moving independent work into parallel execution where possible.
What this lesson answers
- how to estimate agent wall clock latency
- why agent loops get slow with multiple steps
- what parts of an agent can run in parallel
Notes
An agent that takes N sequential steps does not feel like one model call; it feels like the sum of every model call, tool call, database query, retry, and wait in the chain. If each step takes two seconds and the agent must do ten steps in order, the best-case wall-clock is already about twenty seconds before counting overhead. Sequential dependency is the important word: when step 7 cannot start until step 6 finishes, latency multiplies through the loop.
The practical mental model is a critical path diagram. Write down each operation the agent performs, its typical latency, its slow-case…
Common questions
- Why does a multi-step agent feel much slower than a single model call?
- Because the user waits for the longest chain of dependent work. A model call may trigger tool calls, database reads, validation, retries and further reasoning. If each operation depends on the previous result, the delays accumulate rather than overlap, so the experience is closer to the sum of the chain than to any individual call.
- Why are average latencies misleading for agent workflows?
- Averages hide slow outliers, and agent loops give those outliers repeated chances to appear. Even if each individual call is usually fast, a sequence of calls is likely to include at least one slow response. Retries also add spent timeout time before the next attempt, making tail latency central to realistic prediction.
- How do I find what to parallelise in an agent?
- Draw the workflow as dependencies between operations. Anything that does not need the output of another step is a candidate for parallel execution, such as independent retrievals, checks, tool calls or scoring passes. Steps that consume earlier results remain on the critical path, so reducing or removing them usually matters most.
Short definition: what is Latency multiplies in a loop?