01.03 · Concept
Prefill & Decode
Explain the two distinct phases of LLM inference: compute-bound prefill and memory-bound decode. Understand why they must be treated differently.
Prefill builds the attention cache from the whole prompt with parallel, compute-heavy work; decode consumes that cache serially to produce each next token and is usually limited by memory bandwidth. Treating them as separate phases explains first-token latency, per-token latency, batching behaviour and why serving systems optimise them differently.
What this lesson answers
- what is prefill versus decode in LLM inference
- why is LLM decode memory bandwidth bound
- how does KV cache affect generation latency
Notes
Prefill and decode are the two phases of autoregressive LLM inference: prefill processes the input prompt in parallel to build the KV cache, while decode generates one token at a time using that cache. For prompt tokens , prefill computes and logits in one forward pass: , , and stores for every layer.
References
Common questions
- What is the difference between prefill and decode?
- Prefill runs over the input prompt and creates the key-value cache used by attention. It can process prompt positions in parallel, so it tends to look like a large compute job. Decode then generates new tokens one at a time, reusing and extending that cache at every step.
- Why is decode often slower than it looks on paper?
- Each decode step does limited arithmetic for a single new token, but it must read the existing cache and model state needed to attend over prior context. That gives little reuse compared with prefill. The bottleneck is commonly moving data through memory rather than raw floating-point throughput.
- Why should serving systems schedule prefill and decode separately?
- They stress the hardware in different ways and affect different user-visible timings. Prefill mainly controls time to first token, while decode controls the pace of the remaining output. A scheduler that mixes them blindly can let long prompts block interactive generation or waste memory bandwidth during token production.
Short definition: what is Prefill & Decode?