08.03 · Walkthrough
The Decode-Heavy Workload
Redo module 2's arithmetic for a request whose output is two orders of magnitude longer than its input, and show that a reasoning request is almost pure decode, so every memory-bandwidth conclusion applies harder, and every prefill optimisation matters less.
Reasoning-style generation is dominated by decode because the model pays the autoregressive cost for every emitted scratchpad token. When the answer is orders of magnitude longer than the prompt, weight reads, KV growth, batching limits and memory bandwidth drive latency far more than prompt ingestion speed.
What this lesson answers
- why long reasoning outputs are decode bound
- when prefill optimisation stops mattering
- how KV cache growth affects generation latency
Notes
A decode-heavy workload is one where the output length dominates the prompt length , so the request cost is governed by autoregressive one-token steps rather than by the prompt prefill matrix multiply. The accounting identity is , but for “thinking” requests with , the first term is usually a rounding error in wall time and GPU memory traffic.
References
Common questions
- Why does a long reasoning response cost so much more than a normal chat response?
- A normal prompt is processed once during prefill, but every generated token requires another decode step. Long reasoning traces create many more serial decode steps, each involving model weight access and KV cache updates. The cost therefore grows with the number of generated tokens, not just the input size.
- Does faster prefill help much for decode-heavy workloads?
- Only a little. Faster prefill helps when prompt processing is a meaningful part of end-to-end latency. In a decode-heavy request, the prompt cost is paid once and then swamped by repeated generation steps. Improving per-token decode throughput usually saves much more time than making prompt ingestion faster.
- Why does the KV cache become a bottleneck during long generation?
- Each generated token appends keys and values, and later tokens must attend over a longer history. That means memory use grows with the generated sequence, and attention reads become heavier as decoding progresses. Paging and efficient layouts reduce waste, but they do not remove the underlying cache growth.
Short definition: what is Decode-Heavy Workload?