07.10 · Walkthrough
Long-Context Prefill
Take module 4's chunked prefill to agent scale: schedule a hundred-thousand-token prompt without starving every decoding request behind it, and show why the cache hit rate matters more here than any kernel does.
Long-context prefill is primarily a scheduling and KV-cache problem, not an attention-kernel problem. Large agent prompts must be split into bounded chunks, interleaved with decode work, and made cheap through prefix reuse. Otherwise one cache-miss prompt can monopolise GPU time and memory while active conversations wait for tokens.
What this lesson answers
- how does chunked prefill prevent decode starvation
- why does prefix cache hit rate dominate long prefill
- when should agent prefill run on separate workers
Notes
Long-context prefill is the scheduling problem created when an agent submits a very large prompt, often a transcript plus retrieved tool outputs, whose first-token computation is a full forward pass over prompt tokens rather than the one-token decode step used for active conversations. For a decoder-only transformer the dominant per-layer prefill attention work is , with approximate FLOPs , while decode against an existing KV cache is per generated token.
References
Common questions
- Why is long-context prefill different from normal decoding?
- Prefill processes the whole prompt to create the initial KV cache, while decoding usually adds one token against an existing cache. A very large agent prompt therefore creates a large burst of compute and memory allocation before the first generated token appears. If scheduled naively, that burst blocks interactive decode work behind it.
- What does chunked prefill actually improve?
- Chunking does not make the total prompt computation disappear. It limits how long one prefill request can occupy the scheduler before decode work gets another chance to run. That changes the queueing behaviour from one long head-of-line stall into many smaller scheduling quanta that can be interleaved with active conversations.
- Why can prefix caching matter more than faster attention kernels?
- A kernel can reduce the cost of executing attention, but prefix caching can avoid executing large parts of the prompt at all. Agent workloads often reuse system prompts, tool schemas, memory summaries, and stable retrieved context. A high cache hit rate reduces both prefill compute and new KV allocation, which directly protects latency and capacity.
Short definition: what is Long-Context Prefill?