07.06 · Concept
Tool Calls in the Loop
Account for the pause: while a tool runs, the sequence is neither prefilling nor decoding, and its KV cache is occupying memory it is not using. Compare holding, offloading and evicting it, and say which the tool latency distribution argues for.
Tool waits create a suspended serving state: no prefill, no decode, but KV cache may still occupy accelerator memory. The right policy is driven by the tool latency tail: hold for consistently short calls, offload when transfer cost is amortised, and evict when waits are long, abandoned, or externally gated.
What this lesson answers
- why tool calls waste KV cache memory
- when should agent serving offload KV cache
- hold offload or evict KV during tool calls
Notes
A tool call inserts a third state into the usual prefill/decode accounting: after the model emits a structured call, the request is suspended until an external function returns, so its token sequence performs no attention work but its KV pages still reserve HBM unless the serving system moves or discards them. For a request with resident tokens, transformer layers, KV heads, head dimension , and element size bytes, the live KV footprint is , where the factor 2 is K and V.
References
Common questions
- What happens to KV cache while an agent waits for a tool?
- After the model emits a tool call, the request stops doing attention work until the external result returns. Its KV cache may still remain allocated, so it is not contributing tokens to a decode batch but can still block memory needed by active requests. That suspended state needs its own serving policy.
- Should a serving system keep KV cache in GPU memory during tool calls?
- Keep it only when tool latency is tightly bounded and short enough that immediate resume matters more than memory pressure. Holding gives the fastest continuation, but it fails when many calls sit in the tail of the latency distribution and accumulate inactive KV pages that prevent useful decode work.
- How do offloading and eviction differ for paused agent requests?
- Offloading preserves the exact cache but pays a transfer cost when moving it away from and back to accelerator memory. Eviction frees the memory completely, but the prefix must be replayed when the tool returns. Offload suits moderate waits that usually return soon; eviction suits long, fat-tailed, or human-driven waits.
Short definition: what is Tool Calls in the Loop?