02.06 · Concept
Why Decode Is Memory Bound
Prove from FLOP and byte counts that a single decode step reads far more bytes than it computes operations. Explain why larger batches are the only lever that helps.
Autoregressive decode is usually limited by memory bandwidth because each new token streams the model weights again while doing only a small matrix-vector computation. Tensor cores sit underused not because the GEMM is necessarily bad, but because there is too little weight reuse. Larger live batches improve throughput by sharing each weight stream across more sequences.
What this lesson answers
- why is llm decode memory bound
- how batch size improves decode throughput
- why tensor utilisation is low during generation
Notes
In autoregressive decode, the mechanism is a one-token matrix-vector pass whose weights are reread for every generated token while the batch supplies too little reuse to saturate tensor cores. For a dense transformer layer, the dominant weight traffic per decode step is approximately bytes, where is parameter count and is bytes per weight, while the arithmetic is approximately FLOPs for batch size because each weight participates in one multiply-add per sequence.
References
Common questions
- Why is decode different from prefill?
- Prefill processes many prompt tokens at once, so matrix operations have enough shape and reuse to keep compute units busy. Decode produces one new token per sequence, so each layer mostly performs a matrix-vector pass. The weights still have to be read, but there is far less arithmetic per byte moved.
- Does FlashAttention fix memory-bound decode?
- FlashAttention reduces attention memory traffic and is very important for efficient attention kernels, especially during prompt processing. During decode, however, the new query still has to compare against the existing key and value cache. It cannot remove the need to read old cache entries or stream the model weights.
- Why does batching help decode throughput?
- Batching lets multiple sequences use the same weights while they are resident or being streamed through the GPU. The bytes for weights stay roughly fixed for a decode step, while the useful multiply-add work rises with the number of sequences. The limit is reached when cache traffic, latency, and synchronisation costs catch up.