What problem does chunked prefill solve in LLM serving?

It fixes head-of-line blocking caused by admitting a long prompt as one large prefill job. In continuous batching, decode streams need frequent small steps to keep producing tokens. Chunked prefill limits how much prompt work can enter a scheduling turn, letting decode tokens share the same batch.

Does chunked prefill make the long prompt start generating earlier?

No. The request with the long prompt still needs its prompt KV to be fully built before it can decode its own first generated token. The benefit is fairness for other requests: their decode steps continue while the long prompt is processed in pieces.

Why can chunks that are too small be bad?

Prefill is efficient partly because it processes many prompt tokens together. Very small chunks reduce that arithmetic efficiency and add scheduler, kernel launch, CUDA graph, and KV bookkeeping overhead. They can also worsen time to first token for the long-prompt request by spreading its required prefill across more turns.

The Batch

04.05 · Walkthrough

Chunked Prefill

Explain how splitting a long prompt into token-budgeted chunks lets prefill and decode share a batch, so a single long prompt stops stalling every other user's stream.

Chunked prefill splits long prompt processing into bounded scheduler turns, so prompt KV construction can run alongside decode work from existing streams. The long request still cannot generate until its prompt is complete, but other users keep receiving tokens instead of sitting behind one large context-processing job.

What this lesson answers

how does chunked prefill reduce stream stalls
why do long prompts block decode batching
when does chunked prefill hurt latency

Notes

Chunked prefill is the scheduler mechanism that breaks a request’s prompt computation into bounded pieces of at most $C$ tokens, so the batch at each iteration can contain both decode tokens from already-running streams and a slice of prefill tokens from one or more long prompts. Without it, a prefill for prompt length $P$ enters as one monolithic attention/MLP job and blocks decode admission until all $P$ positions have produced KV; with it, the scheduler runs $⌈ P / C ⌉$ prefill micro-steps, each appending KV for a contiguous range.

References

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Common questions

What problem does chunked prefill solve in LLM serving?: It fixes head-of-line blocking caused by admitting a long prompt as one large prefill job. In continuous batching, decode streams need frequent small steps to keep producing tokens. Chunked prefill limits how much prompt work can enter a scheduling turn, letting decode tokens share the same batch.
Does chunked prefill make the long prompt start generating earlier?: No. The request with the long prompt still needs its prompt KV to be fully built before it can decode its own first generated token. The benefit is fairness for other requests: their decode steps continue while the long prompt is processed in pieces.
Why can chunks that are too small be bad?: Prefill is efficient partly because it processes many prompt tokens together. Very small chunks reduce that arithmetic efficiency and add scheduler, kernel launch, CUDA graph, and KV bookkeeping overhead. They can also worsen time to first token for the long-prompt request by spreading its required prefill across more turns.

Short definition: what is Chunked Prefill?

04.04 · The BatchThe Scheduler Loop 04.06 · The BatchPrefill/Decode Interference