Is prefill just the first decode step?

No. That is a common misunderstanding. Prefill processes the prompt tokens as a block and builds the KV cache for the whole prompt. Decode then consumes one newly generated token at a time, using that cache instead of recomputing the previous context. They both run the transformer, but the shapes, bottlenecks, and scheduling choices differ.

Why is decode often slower than it looks computationally?

Per token, decode does much less arithmetic than prompt processing, but it has poor reuse. Each step must consult the accumulated KV cache and model parameters while producing only one new position. That makes memory bandwidth and cache layout more important than raw floating-point throughput, especially for small batches or long contexts.

Which phase should I optimise first?

It depends on the workload. Long prompts with short answers are often dominated by prefill and time-to-first-token. Short prompts with long answers are often dominated by decode and time-per-output-token. Interactive products usually care about both: users notice the first token, then notice the cadence of the remaining tokens.

How do serving systems handle prefill and decode differently?

They usually expose the distinction in scheduling and cache management. Prompt requests may be batched or chunked to keep compute busy without blocking active generations. Decode requests may be continuously batched, use paged KV storage, and rely on kernels tuned for reading cached attention state and appending one token’s KV entries.

Prefill & Decode

Prefill and decode are the two execution stages of autoregressive LLM serving: prefill ingests the whole prompt to initialise attention state, then decode extends the sequence one token at a time. The split matters because prefill mostly stresses compute throughput, while decode mostly stresses memory bandwidth and cache management.

A single “generate” call hides two very different workloads. The prompt can be processed with wide parallel operations, but the answer cannot: each output token depends on the token chosen immediately before it. Treating both as the same kind of forward pass leads to poor scheduling decisions, misleading latency metrics, and optimisations that help time-to-first-token but barely affect the long tail of generation.

In prefill, the server runs the prompt tokens through the transformer together under a causal mask. It computes the hidden states, produces the first next-token logits, and writes each layer’s attention keys and values into a KV cache. In decode, the server feeds back the sampled token, reads the cached keys and values for the existing context, computes only the new token’s state, appends its KV entries, and repeats.

The tradeoff is that prefill and decode want different hardware behaviour. Prefill has large matrix operations with substantial reuse, so it can keep accelerators busy. Decode has tiny per-token work but must repeatedly read model weights and growing KV state, so memory movement dominates. Batching helps both, but not identically: prefill batching can delay first tokens, while decode batching is limited by uneven sequence lengths and cache capacity.

Engineers meet this split in inference schedulers, generation loops, and latency dashboards. Metrics such as time-to-first-token describe prefill plus initial scheduling, while time-per-output-token describes the decode loop. Serving systems often schedule prompt work separately from generation work, use paged KV caches, interleave chunked prompt processing with active decodes, and apply different optimisations to each phase rather than tuning “inference” as one blob.

Go deeper

Inference EngineeringPrefill & DecodeExplain the two distinct phases of LLM inference: compute-bound prefill and memory-bound decode. Understand why they must be treated differently.Open the lesson

Common questions

Is prefill just the first decode step?: No. That is a common misunderstanding. Prefill processes the prompt tokens as a block and builds the KV cache for the whole prompt. Decode then consumes one newly generated token at a time, using that cache instead of recomputing the previous context. They both run the transformer, but the shapes, bottlenecks, and scheduling choices differ.
Why is decode often slower than it looks computationally?: Per token, decode does much less arithmetic than prompt processing, but it has poor reuse. Each step must consult the accumulated KV cache and model parameters while producing only one new position. That makes memory bandwidth and cache layout more important than raw floating-point throughput, especially for small batches or long contexts.
Which phase should I optimise first?: It depends on the workload. Long prompts with short answers are often dominated by prefill and time-to-first-token. Short prompts with long answers are often dominated by decode and time-per-output-token. Interactive products usually care about both: users notice the first token, then notice the cadence of the remaining tokens.
How do serving systems handle prefill and decode differently?: They usually expose the distinction in scheduling and cache management. Prompt requests may be batched or chunked to keep compute busy without blocking active generations. Decode requests may be continuously batched, use paged KV storage, and rely on kernels tuned for reading cached attention state and appending one token’s KV entries.

Every term