Chunked Prefill
Chunked prefill is an LLM serving scheduler technique that splits a long prompt’s prefill work into bounded token chunks, then batches those chunks alongside decode steps from other requests. It prevents one large prompt from monopolising a GPU iteration while its key-value cache is being built.
The problem is head-of-line blocking in continuous batching. Decode work is latency-sensitive because active streams need their next token again and again. Prefill work for a long prompt is a bulky context pass that must compute and store KV for every prompt position. If the scheduler admits that prefill as one large job, other users’ streams can pause even though each only needed a small decode step.
Chunked prefill makes the scheduler treat prompt processing as resumable work. Instead of scheduling all P prompt tokens at once, it schedules chunks of at most C tokens. Each chunk appends the next KV entries for the same sequence, attends to the earlier prompt KV plus the current chunk under a causal mask, then yields. A batch iteration can therefore contain D one-token decodes and some prefill chunks, as long as the total fits the engine’s token budget T.
A common misunderstanding is that chunked prefill lets the same request start generating before its prompt is processed. It does not. That request still cannot decode until its own prompt KV is complete. The benefit is fairness across requests. Existing streams keep their one-token cadence while the long prompt advances over multiple scheduler turns, rather than forcing everyone to wait behind one monolithic context computation.
The trade-off is that smaller chunks are not free. Prefill is efficient partly because it uses wide matrix operations over many tokens; over-splitting can reduce arithmetic intensity and add scheduler overhead, KV page bookkeeping, graph complications, and extra attention launches. It can also worsen time to first token for the long-prompt request, because interleaving decode work intentionally delays completion of its full prefill. Whether it helps depends on load, latency goals, chunk size, and available batching slack.
Engineers usually meet chunked prefill in inference servers and schedulers, not in model code. It appears alongside continuous batching, paged KV-cache allocation, in-flight batching, and prefix-cache systems. The implementation is about token budgets and KV pages, not splitting text semantically. FlashAttention-style kernels may make each chunk fast, but the important policy decision is whether prefill may consume only its allotted chunk in a scheduling turn.
Common questions
- How is chunked prefill different from normal batching?
- Normal batching can place requests together, but a long prefill may still enter the batch as one large unit. Chunked prefill changes the unit of scheduling: prompt computation is broken into bounded pieces. That lets the scheduler mix a slice of context processing with decode tokens from active streams in the same iteration.
- Does chunked prefill reduce KV-cache memory?
- No. The full prompt still needs KV entries before that request can generate. Chunking changes when the KV is produced, not how much final KV is required. It can make allocation and admission smoother when paired with paged KV storage, but it is not a memory compression technique.
- When should chunked prefill be avoided or tuned carefully?
- It depends on the workload. If there is little decode backlog, chunking may only add overhead. If chunks are too small, prefill becomes less efficient and launch or scheduling costs can dominate. If chunks are too large, decode streams still see stalls. The useful setting is usually workload-specific.
- Is chunked prefill the same as streaming the prompt?
- No. Streaming output means returning generated tokens to the user as they are produced. Chunked prefill is internal scheduling for prompt processing. The long-prompt request cannot emit its first generated token until all of its prompt chunks have run and its prompt KV is complete.