How is a decode-heavy workload different from a long-context workload?

A long-context workload has a large prompt, so prefill and attention over the input can be expensive. A decode-heavy workload has a large output, so the expensive part is many serial generation steps. They can overlap, but they stress systems differently: long context stresses prompt processing and KV footprint early, while decode-heavy traffic stresses per-token bandwidth and scheduling for the whole response.

Why does memory bandwidth matter so much during decode?

Each generated token requires another forward pass through the decoder. For small per-request decode batches, the accelerator often cannot reuse weights enough to become compute-bound, so it spends much of its time streaming model weights and reading KV state from high-bandwidth memory. As the sequence grows, KV reads add more pressure, making late tokens increasingly costly.

Do FlashAttention or paged KV caches fix decode-heavy inference?

They help, but they do not change the basic shape of the problem. FlashAttention is very important for efficient prompt and chunked prefill, and paged KV layouts reduce fragmentation and make batching easier. In a decode-heavy request, however, the system still has to perform one dependent decode step for each generated token, with growing KV state.

Can speculative decoding make a decode-heavy workload cheap?

Only when the draft model is cheap and the target model accepts enough proposed tokens. If the task has unstable reasoning branches, strict formatting, tool constraints, or exploratory sampling, acceptance can fall and the draft path becomes overhead. Speculation reduces target-model passes when predictions are locally reliable; it does not remove the sequential dependency between accepted tokens.

Decode-Heavy Workload

A decode-heavy workload is an inference request where generated output is far longer than the input prompt, so latency and GPU traffic are dominated by repeated autoregressive decode steps. It is common in reasoning traces: prefill is paid once, but every produced token rereads weights, extends KV cache, and attends over growing context.

The need for this term comes from a misleading habit: treating a request as if prompt processing and answer generation were comparable costs. They are not when the model writes a long scratchpad or reasoning trace. Prefill is the model ingesting the prompt in a relatively parallel pass. Decode is serial token production. When output is two orders of magnitude longer than input, the request is almost all decode, so prompt-side improvements barely move end-to-end latency.

Mechanically, the request cost is prefill plus one decode iteration per output token. In each decode step, the model consumes the previous token, rereads its weights, computes the next-token distribution, appends new keys and values to the KV cache, and samples or selects a token. The cache also grows with the sequence, so later steps have more attention state to read. This makes memory bandwidth, not raw arithmetic throughput, the first limit engineers should picture.

The trade-off is that long reasoning is not free internal computation. It is bought as visible or hidden generated tokens, each charged at decode price. Techniques such as paged KV caches, continuous batching, fused kernels, FlashAttention, and speculative decoding can improve constants, but none remove the causal chain from one token to the next. A common misunderstanding is that faster prefill solves long reasoning latency. It depends on the ratio, and in decode-heavy traffic prefill is usually a small term.

Engineers meet decode-heavy workloads in LLM serving for chain-of-thought-style answers, tool-free planning, code generation, long JSON emission, and agents that produce many intermediate tokens. It shows up in metrics such as inter-token latency, decode throughput, KV-cache memory, batch occupancy, and output length distributions. In practice, systems like vLLM, SGLang, TensorRT-LLM, Dynamo-style disaggregation, and Kubernetes-native serving stacks become decode schedulers first and prompt accelerators second for this traffic.

Go deeper

Inference EngineeringThe Decode-Heavy WorkloadRedo module 2's arithmetic for a request whose output is two orders of magnitude longer than its input, and show that a reasoning request is almost pure decode, so every memory-bandwidth conclusion applies harder, and every prefill optimisation matters less.Open the lesson

Common questions

How is a decode-heavy workload different from a long-context workload?: A long-context workload has a large prompt, so prefill and attention over the input can be expensive. A decode-heavy workload has a large output, so the expensive part is many serial generation steps. They can overlap, but they stress systems differently: long context stresses prompt processing and KV footprint early, while decode-heavy traffic stresses per-token bandwidth and scheduling for the whole response.
Why does memory bandwidth matter so much during decode?: Each generated token requires another forward pass through the decoder. For small per-request decode batches, the accelerator often cannot reuse weights enough to become compute-bound, so it spends much of its time streaming model weights and reading KV state from high-bandwidth memory. As the sequence grows, KV reads add more pressure, making late tokens increasingly costly.
Do FlashAttention or paged KV caches fix decode-heavy inference?: They help, but they do not change the basic shape of the problem. FlashAttention is very important for efficient prompt and chunked prefill, and paged KV layouts reduce fragmentation and make batching easier. In a decode-heavy request, however, the system still has to perform one dependent decode step for each generated token, with growing KV state.
Can speculative decoding make a decode-heavy workload cheap?: Only when the draft model is cheap and the target model accepts enough proposed tokens. If the task has unstable reasoning branches, strict formatting, tool constraints, or exploratory sampling, acceptance can fall and the draft path becomes overhead. Speculation reduces target-model passes when predictions are locally reliable; it does not remove the sequential dependency between accepted tokens.

Every term