Is Paged Attention the same thing as FlashAttention?

No. FlashAttention is mainly a kernel technique for doing attention with better use of fast on-chip memory and less unnecessary traffic. Paged Attention is a serving-time KV-cache layout and allocator. They are commonly used together: FlashAttention improves the computation path, while Paged Attention decides where each request’s cached keys and values live.

Why is it described as a baseline rather than an optimisation?

Because high-throughput LLM serving is usually limited by KV-cache memory management, not just raw model execution. Without block-managed KV, the server wastes memory through over-reservation and fragmentation, which reduces batching and causes avoidable rejections. Paged Attention is now the expected foundation for serious dynamic batching, prefix caching, and long-running multi-request inference.

Does Paged Attention eliminate all KV-cache waste?

No. It bounds the main waste to the unused tail of each sequence’s final block, plus block-table metadata and allocator overhead. The amount depends on block size and workload shape. Short, bursty chats favour smaller blocks; very small blocks can hurt kernel efficiency. The right setting is a trade-off, not a universal constant.

What happens when the GPU runs out of KV blocks?

The runtime must choose a policy. It can delay or reject new requests, evict cached prefixes, preempt sequences, move blocks to CPU memory, or discard and later recompute KV from tokens. The honest answer is workload-dependent. Swapping can preserve progress, but it often makes per-token latency unpredictable because KV blocks are large and decode is latency-sensitive.

Paged Attention

Paged Attention is a KV-cache memory layout for LLM serving that stores each request’s cached keys and values in fixed-size blocks and uses a per-request block table to translate logical sequence positions into physical memory blocks, so attention can read a continuous context while the allocator reuses scattered GPU memory.

LLM servers keep a KV cache for every live sequence, and that cache often consumes the memory that determines how many requests can run together. A naive contiguous allocator must reserve space for a possible maximum length, even though many requests finish early. As requests arrive and complete, holes of awkward sizes appear. The result is wasted GPU memory and unnecessary admission failures, not because arithmetic capacity is exhausted, but because the cache cannot be packed well.

Paged Attention fixes this by treating the KV cache like virtual memory. Each sequence has logical blocks for its token positions. The runtime owns a pool of physical KV blocks. A block table maps a sequence’s logical block index to the physical block currently holding those keys and values. During attention, the kernel computes the token’s logical block and offset, looks up the physical block id, then loads the KV data from that block without first copying the sequence into contiguous memory.

The trade-off is another level of indirection in the attention kernel. Very small blocks reduce tail waste but create more table lookups and less regular memory access. Very large blocks make the table cheaper but bring back unused space at the end of many short sequences. Paged Attention also does not make long-context attention cheap. It solves cache placement and fragmentation, while the model may still need to read or process a long logical history.

Engineers usually meet Paged Attention through vLLM-style serving stacks, paged KV-cache settings, block managers, prefix caches, and schedulers for continuous batching. During prefill, blocks are allocated and filled as prompt tokens produce K and V. During decode, new tokens append into the current tail block or trigger another allocation. When a request finishes, its blocks return to the free list. Under memory pressure, systems may evict, swap, recompute, or reject work.

Go deeper

Inference EngineeringPaged AttentionDesign paged attention: map logical KV blocks to physical ones via a block table, eliminate fragmentation, and explain why this is the 2026 floor, not an optimisation. Trace vLLM's block-level memory management from allocation through eviction.Open the lesson

Common questions

Is Paged Attention the same thing as FlashAttention?: No. FlashAttention is mainly a kernel technique for doing attention with better use of fast on-chip memory and less unnecessary traffic. Paged Attention is a serving-time KV-cache layout and allocator. They are commonly used together: FlashAttention improves the computation path, while Paged Attention decides where each request’s cached keys and values live.
Why is it described as a baseline rather than an optimisation?: Because high-throughput LLM serving is usually limited by KV-cache memory management, not just raw model execution. Without block-managed KV, the server wastes memory through over-reservation and fragmentation, which reduces batching and causes avoidable rejections. Paged Attention is now the expected foundation for serious dynamic batching, prefix caching, and long-running multi-request inference.
Does Paged Attention eliminate all KV-cache waste?: No. It bounds the main waste to the unused tail of each sequence’s final block, plus block-table metadata and allocator overhead. The amount depends on block size and workload shape. Short, bursty chats favour smaller blocks; very small blocks can hurt kernel efficiency. The right setting is a trade-off, not a universal constant.
What happens when the GPU runs out of KV blocks?: The runtime must choose a policy. It can delay or reject new requests, evict cached prefixes, preempt sequences, move blocks to CPU memory, or discard and later recompute KV from tokens. The honest answer is workload-dependent. Swapping can preserve progress, but it often makes per-token latency unpredictable because KV blocks are large and decode is latency-sensitive.

Every term