Is a tool-wait just lower GPU utilisation?

Not exactly. The important effect is not only that the request stops consuming compute. It may also keep its KV cache allocated while absent from the decode batch. That creates memory pressure and admission failures for other requests, even when the accelerator has compute capacity available.

Why not always keep the KV cache in memory?

Keeping it is best when the tool returns quickly and predictably. It becomes expensive when many requests wait behind slow or variable tools, because each paused sequence holds memory that active decoders could use. The decision depends on cache size, available HBM, traffic burstiness, and the tool latency tail.

Is offloading better than evicting?

It depends on whether copying the cache out and back is cheaper than recomputing it. Offload keeps the continuation exact and avoids replay, but adds transfer delay and contention. Eviction is attractive for very long or uncertain waits, but poor for short calls because it forces another prefill before decoding can continue.

Tool Calls in the Loop

Tool calls in the loop are pauses in an agent inference request where the model has emitted a structured external call, the application is waiting for that call to return, and the request is doing no prefill or decode work while its accumulated KV cache may still occupy accelerator memory.

The problem is that agent traffic is not a continuous stream of model computation. A request can decode until it asks for an API call, database lookup, or other function, then sit idle while that external work runs. During that wait it has left the decode batch, so it is not using attention compute, but its previous tokens may still reserve KV cache pages. Enough paused conversations can block active requests even though the GPU looks underused.

Mechanically, the request moves through prefill, decode, tool-wait, then resume. At the call boundary, the serving system has a complete prefix: prompt, conversation, and the model’s tool-call tokens. That prefix has key and value tensors for every layer. The scheduler must decide whether those pages stay in HBM, are copied to host or storage and copied back later, or are freed and rebuilt by replaying the transcript when the tool result arrives.

Holding the KV cache gives the fastest resume, because the request can rejoin decoding without replaying its prefix. The cost is memory stranded behind an external dependency. Offloading preserves exact continuation but adds transfer latency, host memory pressure, and possible contention with other GPU work. Eviction frees scarce memory, but turns the return path into another prefill. The common mistake is planning from average tool latency; the tail determines how many paused caches accumulate.

Engineers meet this in vLLM-style paged KV systems, TensorRT-LLM cache management, disaggregated serving stacks, and agent runtimes that support structured tool calls. The policy usually depends on the observed latency distribution: hold for tightly bounded low-latency RPCs, offload when waits are long enough to justify copying but likely to return soon, and evict for human pauses, webhooks, queues, or fat-tailed tools where memory would otherwise be pinned indefinitely.

Go deeper

Inference EngineeringTool Calls in the LoopAccount for the pause: while a tool runs, the sequence is neither prefilling nor decoding, and its KV cache is occupying memory it is not using. Compare holding, offloading and evicting it, and say which the tool latency distribution argues for.Open the lesson

Common questions

Is a tool-wait just lower GPU utilisation?: Not exactly. The important effect is not only that the request stops consuming compute. It may also keep its KV cache allocated while absent from the decode batch. That creates memory pressure and admission failures for other requests, even when the accelerator has compute capacity available.
Why not always keep the KV cache in memory?: Keeping it is best when the tool returns quickly and predictably. It becomes expensive when many requests wait behind slow or variable tools, because each paused sequence holds memory that active decoders could use. The decision depends on cache size, available HBM, traffic burstiness, and the tool latency tail.
Is offloading better than evicting?: It depends on whether copying the cache out and back is cheaper than recomputing it. Offload keeps the continuation exact and avoids replay, but adds transfer delay and contention. Eviction is attractive for very long or uncertain waits, but poor for short calls because it forces another prefill before decoding can continue.

Every term