Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al.2023SOSP 2023
Read it on arxiv.org(opens in a new tab)Why this one
Read this after FlashAttention and Efficiently Scaling Transformer Inference. Those papers teach you that attention is as much an IO problem as a math problem. PagedAttention makes the next jump: in production serving, the scarce thing is not just FLOPs, it is the messy, growing KV cache owned by thousands of half-finished requests. The trick is wonderfully old: treat KV blocks like virtual memory pages, stop reserving giant contiguous buffers, and let the scheduler batch around real occupancy instead of worst-case guesses. People often describe vLLM as “faster attention,” but the durable idea is memory management as a serving primitive. If you are building an LLM endpoint, this paper changes what you measure: not tokens per second in a clean loop, but wasted cache, prefix sharing, preemption, and batch shape under live traffic.
What to take away
- The KV cache is the dominant mutable state in autoregressive serving, and naive contiguous allocation wastes expensive GPU memory.
- PagedAttention splits each sequence cache into fixed-size blocks, letting requests grow without large reservations or copying.
- The system payoff is bigger batches at the same latency, especially for long prompts, beam search, and shared prefixes.
Reads with
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
sets up attention as an io bound memory movement problem
- Efficiently Scaling Transformer Inference
frames decoding throughput around batching and kv cache pressure
- Fast Transformer Decoding: One Write-Head is All You Need
attacks the same kv cache bottleneck by shrinking what each token must store