Is the memory wall the same as running out of GPU memory?

Not exactly. Running out of memory is one failure mode, but the wall also includes bandwidth. A request may fit in HBM yet decode slowly because each new token must stream a large KV cache. Capacity decides how many long contexts can be resident; bandwidth decides how quickly they can be used.

Why does long context cost so much if the model weights are unchanged?

Because the provider is not only running the same model. It is reserving KV-cache memory proportional to the active context and repeatedly reading it during generation. The weights are mostly fixed per replica, while KV cache grows with each user’s prompt, batch position, and generated history. Long context rents scarce HBM.

Do PagedAttention and prefix caching solve the memory wall?

They help with allocation and reuse, not with the fundamental size of a unique long prompt. PagedAttention stores KV blocks through a page-table-like structure, improving packing and dynamic batching. Prefix caching can share identical prompt prefixes. If requests have unrelated long documents, the same bytes still need to be stored and streamed.

Does speculative decoding bypass the KV-cache bottleneck?

It depends on the bottleneck. Speculative decoding can reduce expensive target-model calls by verifying draft tokens in groups, which helps when model invocation or weight traffic dominates. At very long context, the verifier still attends over the resident KV cache, and the draft model may need its own cache too.

Memory Wall

The memory wall is the point in long-context LLM inference where serving is limited more by KV-cache capacity and memory bandwidth than by arithmetic or model weights. Past that point, each generated token requires storing and streaming so much prior attention state that context length becomes an infrastructure cost, not just a model capability.

Autoregressive models need the past to produce the next token. Without a cache, every decode step would repeatedly recompute attention projections for the whole prompt and generated history. The KV cache avoids that waste, but it creates a different bottleneck: all previous keys and values must stay available while generation continues. At long context, the active request is no longer mainly a compute job. It is a large resident memory allocation that must be read again and again.

Concretely, each layer stores key and value tensors for every token in the resident context, across the KV heads and head dimension, using the chosen cache precision. On each new token, the server appends one new key and value row, then attention reads the relevant cached rows to compare the new query against the history. The footprint grows with batch size, context length, layer count, KV-head count, head size, and bytes per element.

The trade-off is that caching saves recomputation but consumes scarce high-bandwidth memory and sustained memory traffic. FlashAttention can make attention kernels use on-chip memory better, and paged KV layouts can reduce fragmentation, but neither removes the cache. Offloading to CPU or disk may be cheaper than recomputing for some prefill or reuse cases, but it can be disastrous for decode if every generated token needs those bytes immediately.

Engineers meet the memory wall when advertised long context fails to translate into cheap, high-throughput serving. Around 32K context and above, admission control, batching, prefix reuse, cache eviction, and prefill/decode separation often matter more than raw FLOPs. This is why APIs price long inputs differently and why serving stacks such as vLLM, SGLang, TensorRT-LLM-style runtimes, Dynamo, and llm-d treat KV placement as a first-class scheduling problem.

Go deeper

Inference EngineeringThe Memory WallExplain why KV cache memory dominates inference cost above ~32K context, reframing long-context serving as a memory product rather than a model capability. Understand why context length is priced the way it is.Open the lesson

Common questions

Is the memory wall the same as running out of GPU memory?: Not exactly. Running out of memory is one failure mode, but the wall also includes bandwidth. A request may fit in HBM yet decode slowly because each new token must stream a large KV cache. Capacity decides how many long contexts can be resident; bandwidth decides how quickly they can be used.
Why does long context cost so much if the model weights are unchanged?: Because the provider is not only running the same model. It is reserving KV-cache memory proportional to the active context and repeatedly reading it during generation. The weights are mostly fixed per replica, while KV cache grows with each user’s prompt, batch position, and generated history. Long context rents scarce HBM.
Do PagedAttention and prefix caching solve the memory wall?: They help with allocation and reuse, not with the fundamental size of a unique long prompt. PagedAttention stores KV blocks through a page-table-like structure, improving packing and dynamic batching. Prefix caching can share identical prompt prefixes. If requests have unrelated long documents, the same bytes still need to be stored and streamed.
Does speculative decoding bypass the KV-cache bottleneck?: It depends on the bottleneck. Speculative decoding can reduce expensive target-model calls by verifying draft tokens in groups, which helps when model invocation or weight traffic dominates. At very long context, the verifier still attends over the resident KV cache, and the draft model may need its own cache too.

Every term