The Agent Workload

Notes

Agent serving is the regime where one logical user request becomes a trajectory of LLM calls over a growing transcript, tool results, retrieved documents, and scratch state. For turn $i$, the prompt is usually $P_i=S\Vert h_1\Vert o_1\Vert \cdots \Vert h_{i-1}\Vert o_{i-1}\Vert u_i$, where $S$ is the system/developer prefix, $h_j$ are prior model messages and $o_j$ are observations from tools. The important quantity is not request count but prefix overlap: ${\rho }_i=\frac{\mathrm{LCP}(P_i,P_{i-1})}{|P_i|}$, often above $0.9$ for ReAct-style agents because each turn appends a small delta to a long transcript. The input/output ratio also flips: $r=\frac{{\sum }_i|P_i|}{{\sum }_i|Y_i|}$ can be $10$-$100$, so the workload is dominated by prefill and prefix lookup rather than by long autoregressive decode. This is the traffic shape that invalidates the ordinary chat-serving picture used for single-turn prompts with modest context and little cross-request sharing.

A concrete H100 example shows why this matters. Take a 70B FP16 model on an 80GB H100 with $3.35$ TB/s HBM bandwidth, grouped-query attention with $8$ KV heads and head dim $128$, and assume $80$ layers. The KV cache footprint per token is $2\cdot L\cdot H_{kv}\cdot d\cdot 2$ bytes $=2\cdot 80\cdot 8\cdot 128\cdot 2=327,680$ bytes, about $320$ KiB/token. A $32$k-token agent transcript therefore has about $32,768\cdot 327,680=10.74$ GB of KV. If the next agent step appends only $512$ new tokens before generating $128$ tokens, recomputing the whole prompt costs prefill over $33,280$ tokens, while exact prefix reuse prefills only $512$. Even ignoring FLOPs and looking only at KV residency, the cached prefix is $10.74$ GB that must be retained or paged, not recomputed. This is why PagedAttention from the vLLM paper and automatic prefix caching in vLLM, SGLang’s RadixAttention paper, and TensorRT-LLM’s KV cache reuse features are central to agent serving rather than optional latency polish.

The first assumption from a decode-centric module that stops holding is that steady-state generation is the bottleneck and prefill is an amortized setup phase. For normal chat, one might model per-token decode bandwidth as roughly $B_{\mathrm{decode}}\approx \frac{\mathrm{model}\mathrm{bytes}+\mathrm{KV}bytes/read}{\mathrm{token}}$ and optimize batching around many requests producing one token each. In an agent loop, the expensive event is often “same 30k tokens plus a 200-token observation, again,” and the useful batch contains heterogeneous prefills, prefix-cache hits, cache misses, tool-delayed resumes, and short decodes. FlashAttention, from Dao et al., still matters inside prefill because it reduces HBM traffic for attention, but it does not by itself exploit that $P_i$ and $P_{i-1}$ are nearly identical. PagedAttention’s block table, SGLang’s radix tree over token prefixes, TensorRT-LLM’s paged KV cache, NVIDIA Dynamo’s disaggregated serving stack, and llm-d’s Kubernetes-native inference scheduling are responses to this changed bottleneck: route and place requests so the already-materialized prefix is on the worker that needs it.

The second broken assumption is that requests are independent samples from a prompt distribution. Agent turns are causally linked, externally paced by tools, and cache-affine: moving turn $i+1$ to a different GPU can turn a $512$-token incremental prefill into a $33$k-token full prefill or a multi-GB KV transfer. With the numbers above, transferring the $10.74$ GB prefix KV over a $900$ GB/s NVLink path has a lower bound of $10.74/900=11.9$ ms, while over a $64$ GB/s PCIe path it is $168$ ms before software overheads; recomputing on an H100 may also be costly but can beat remote fetch if the link is weak or the prefix is cold. This is the serving problem targeted by prefill/decode disaggregation and cache-aware routing in systems such as NVIDIA Dynamo and llm-d, while vLLM and TensorRT-LLM expose the underlying paged-cache machinery. The scheduler’s unit becomes a trajectory with locality constraints, not an isolated request that can be freely load-balanced to the shortest queue.

The third assumption that stops holding is that batching similar decode lengths is the main lever for utilization. Agent traffic has bursty fan-out, tool gaps, and tiny generations: a planner may emit $40$ tokens, call search, append a $4$k-token page, emit $20$ tokens, call Python, then append a traceback. Large continuous batches can be poisoned by long prefills arriving in the middle, while pure decode workers can starve during tool waits. Speculative decoding with EAGLE, from the EAGLE paper on feature-level extrapolation for lossless acceleration, helps only on the generation slices; if the trajectory spends $85\%$ of GPU time in uncached prefill or cache migration, a $2\times$ decode speedup caps end-to-end gain at $1/(0.85+0.15/2)=1.08\times$. Conversely, when prefix hits are high and each turn emits hundreds of tokens, EAGLE or TensorRT-LLM speculative decoding can again matter. The workload mix, not the model alone, decides whether decode optimizations are visible in user latency.

Prefix caching and cache-affine serving also have sharp failure regimes. They stop helping when prompts are not byte/token identical after the shared prefix: timestamps, randomized tool IDs, nondeterministic JSON key order, per-turn retrieval inserted near the beginning, or changing system instructions can reduce $\mathrm{LCP}$ enough that the radix or hash cache misses. They can make throughput worse when memory reserved for cold prefixes evicts hot active KV, when multi-tenant isolation prevents sharing, or when a scheduler pins trajectories to overloaded GPUs to preserve locality while idle GPUs sit elsewhere. For the 70B example, an 80GB H100 cannot hold model weights of about $140$ GB in FP16 on one device, so tensor parallelism is mandatory; the $10.74$ GB prefix KV is then sharded across ranks, and cache movement becomes a collective placement problem rather than a local map lookup. The agent-serving lesson is therefore specific: long multi-turn trajectories, input-heavy turns, and near-total prefix overlap break the independent, decode-heavy, freely batchable assumptions, replacing them with KV locality, prefix identity, and tool-paced scheduling as first-order constraints.