How is agent workload different from normal chat workload?

Normal chat serving often assumes many independent prompts, each with a prefill phase and then token-by-token decode. Agent workload has linked turns over a growing transcript. The same prefix is sent again and again, outputs are often small, and latency depends heavily on whether the existing key-value cache can be reused.

Why does prefix caching matter so much for agents?

Because most turns begin with nearly the same tokens as the previous turn. If the server has already computed attention state for that prefix, it can reuse the cached key-value blocks and prefill only the newly appended text. Without reuse, the system repeatedly pays for processing the full transcript.

Does speculative decoding solve agent serving latency?

It depends on where time is being spent. Speculative decoding helps the generation slices, especially when turns produce longer outputs. It does little for uncached prefill, cache transfers, or tool-delayed resumes. In many agent workloads, prefix identity, cache placement, and scheduling dominate before decode speed becomes visible.

What commonly breaks prefix reuse in agent systems?

Prefix reuse fails when shared text is not token-identical. Common causes include per-turn timestamps, random tool identifiers, nondeterministic serialisation, changing system prompts, and inserting retrieved documents before older context. Even semantically equivalent prompts can miss the cache if their token sequence differs before the intended shared prefix ends.

Agent Workload

Agent workload is the serving pattern created when one user task becomes a long sequence of LLM calls over an ever-growing transcript, tool outputs, retrieved context, and state. Its defining shape is huge repeated input prefixes, relatively short outputs, and turns that are linked by cache locality rather than independent requests.

The problem is that ordinary chat-serving assumptions stop matching the traffic. A single-turn prompt can often be treated as an independent request with a modest prefill followed by decode. An agent instead resends almost the same transcript on each step, with a small new observation or instruction appended. The input can be an order of magnitude larger than the output, so repeatedly processing the shared prefix becomes the dominant cost.

Concretely, each turn’s prompt is the system and developer context, all earlier model messages, all tool observations, and the latest user or planner input concatenated together. The next turn usually has the same beginning and a small suffix. Serving systems exploit this by keeping the key-value cache for the shared prefix and only prefilling the new suffix, using paged cache blocks, prefix hashes, or radix-tree-style prefix lookup.

The trade-off is that prefix reuse is valuable only when the prefix is actually identical at the token level and still resident where it is needed. Timestamps, unstable JSON ordering, changing instructions, or inserting retrieval near the start can destroy reuse. Keeping old prefixes also consumes scarce memory, and routing a trajectory back to its cached worker may conflict with simple load balancing.

Engineers meet agent workload when deploying ReAct-style agents, tool-using assistants, coding agents, retrieval-heavy workflows, and planner-executor loops. The scheduler’s unit is no longer just a request in a queue. It is a trajectory with tool pauses, cache affinity, short decodes, and long heterogeneous prefills. Optimisations such as prefix caching, paged attention, prefill/decode separation, and cache-aware routing become central.

Go deeper

Inference EngineeringThe Agent WorkloadDescribe the shape of agent traffic: long multi-turn trajectories, input sequences an order of magnitude larger than the output, near-total prefix overlap between turns. Name which specific assumptions from modules 3 and 4 stop holding.Open the lesson

Common questions

How is agent workload different from normal chat workload?: Normal chat serving often assumes many independent prompts, each with a prefill phase and then token-by-token decode. Agent workload has linked turns over a growing transcript. The same prefix is sent again and again, outputs are often small, and latency depends heavily on whether the existing key-value cache can be reused.
Why does prefix caching matter so much for agents?: Because most turns begin with nearly the same tokens as the previous turn. If the server has already computed attention state for that prefix, it can reuse the cached key-value blocks and prefill only the newly appended text. Without reuse, the system repeatedly pays for processing the full transcript.
Does speculative decoding solve agent serving latency?: It depends on where time is being spent. Speculative decoding helps the generation slices, especially when turns produce longer outputs. It does little for uncached prefill, cache transfers, or tool-delayed resumes. In many agent workloads, prefix identity, cache placement, and scheduling dominate before decode speed becomes visible.
What commonly breaks prefix reuse in agent systems?: Prefix reuse fails when shared text is not token-identical. Common causes include per-turn timestamps, random tool identifiers, nondeterministic serialisation, changing system prompts, and inserting retrieved documents before older context. Even semantically equivalent prompts can miss the cache if their token sequence differs before the intended shared prefix ends.

Every term