Is long-context prefill the same as chunked prefill?

No. Chunked prefill is one mechanism used to serve long-context prefill safely. Long-context prefill is the broader scheduling and memory problem created by very large prompts. A robust design also needs paged KV allocation, prefix caching, batching policy, admission control, and sometimes separate placement for prefill-heavy and decode-heavy work.

Why does prefix-cache hit rate matter so much?

A cache hit means the runtime can reuse KV blocks for an already processed prefix instead of running those tokens through prefill again. For agent workloads, many requests share system prompts, schemas, summaries, and stable history. When those prefixes hit, both compute and fresh KV allocation fall. That can dominate any improvement from a faster attention kernel.

Does chunking make long prompts cheap?

No. Chunking mainly changes queueing behaviour. It prevents one long prompt from monopolising the GPU until its full prefill completes, but the model still has to process uncached prompt tokens and store their KV state. If memory is full, prefixes do not overlap, or chunks are evicted before reuse, chunking may only spread the cost out.

When should prefill and decode be separated?

It depends on the request mix, latency target, cache pressure, and hardware layout. Separation helps when long prompts are common enough to disturb interactive decode latency, or when prefill can be batched on workers optimised for prompt processing while decode workers protect token responsiveness. For mostly short prompts, the extra routing and cache movement may not be worth it.

Long-Context Prefill

Long-context prefill is the inference scheduling work needed to process a very large prompt before the first generated token, without letting that prompt block ongoing decode traffic. It usually combines chunked prefill, paged KV cache allocation, prefix caching, and careful batching so long agent contexts are admitted in interruptible pieces.

The problem appears when agent requests stop looking like short chat turns. A prompt may include the system instructions, conversation history, tool schemas, retrieved documents, traces, and tool outputs. Before the model can emit the first new token, it must run the whole prompt through the transformer and create KV cache entries. That prefill phase is much heavier than a normal one-token decode step, so a single long request can create head-of-line blocking for every active conversation waiting for its next token.

A long-context prefill scheduler breaks the prompt into token chunks and interleaves those chunks with decode microbatches. The point is not that each chunk is magically cheaper than the whole prompt; it is that the scheduler creates preemption points. Decode work is usually prioritised, and any remaining token budget is filled with prefill chunks. Paged KV storage makes this practical because a partially processed prompt can keep its cache blocks without needing one contiguous allocation or copying a huge cache region.

The main trade-off is that chunking controls latency, not capacity. The KV cache for a long prompt can still consume a large fraction of available memory, and many simultaneous cache-miss agents can exhaust it. Smaller chunks give finer scheduling but add launch overhead, scheduler work, and sometimes worse kernel shapes. This is why prefix-cache hit rate often matters more than attention-kernel choice: reused prefixes avoid both compute and new KV allocation, while a faster kernel still has to process missed tokens.

Engineers meet long-context prefill in serving runtimes and inference stacks, not in model training code. In practice it shows up as settings for chunked prefill, continuous batching, paged KV cache, automatic prefix caching, context chunking, and sometimes separate prefill and decode worker pools. The common misunderstanding is to treat it as a FlashAttention problem. Better kernels help, but the harder production question is queueing and cache reuse: who gets the GPU now, and which prompt tokens can be skipped because their KV already exists?

Go deeper

Inference EngineeringLong-Context PrefillTake module 4's chunked prefill to agent scale: schedule a hundred-thousand-token prompt without starving every decoding request behind it, and show why the cache hit rate matters more here than any kernel does.Open the lesson

Common questions

Is long-context prefill the same as chunked prefill?: No. Chunked prefill is one mechanism used to serve long-context prefill safely. Long-context prefill is the broader scheduling and memory problem created by very large prompts. A robust design also needs paged KV allocation, prefix caching, batching policy, admission control, and sometimes separate placement for prefill-heavy and decode-heavy work.
Why does prefix-cache hit rate matter so much?: A cache hit means the runtime can reuse KV blocks for an already processed prefix instead of running those tokens through prefill again. For agent workloads, many requests share system prompts, schemas, summaries, and stable history. When those prefixes hit, both compute and fresh KV allocation fall. That can dominate any improvement from a faster attention kernel.
Does chunking make long prompts cheap?: No. Chunking mainly changes queueing behaviour. It prevents one long prompt from monopolising the GPU until its full prefill completes, but the model still has to process uncached prompt tokens and store their KV state. If memory is full, prefixes do not overlap, or chunks are evicted before reuse, chunking may only spread the cost out.
When should prefill and decode be separated?: It depends on the request mix, latency target, cache pressure, and hardware layout. Separation helps when long prompts are common enough to disturb interactive decode latency, or when prefill can be batched on workers optimised for prompt processing while decode workers protect token responsiveness. For mostly short prompts, the extra routing and cache movement may not be worth it.

Every term