Is memory hierarchy just another name for cache?

No. Caches are part of the hierarchy, but the hierarchy also includes explicitly managed storage such as shared memory and registers. The distinction matters on GPUs because high-performance kernels often do not merely hope the cache works; they stage tiles, control reuse, and manage register pressure so data reaches compute units at the required rate.

Why does memory hierarchy matter so much for LLM inference?

Decode often has low arithmetic intensity: each new token may stream large weight matrices and read a growing KV cache while doing relatively little reuse. In that regime, tensor cores are not the limiting resource. Latency is set by bandwidth and locality, so arranging data movement can matter more than adding nominal compute capacity.

Does putting data in shared memory always make a kernel faster?

No. Shared memory helps when many threads reuse the staged data before it is evicted or overwritten. It can hurt if the tile is not reused, if accesses conflict across banks, or if the extra staging reduces occupancy. The honest answer depends on reuse, access pattern, register use, and whether memory latency was actually the bottleneck.

What are register spills and why are they bad?

A spill happens when a thread has more live values than can fit in physical registers, so some values are written to local memory. Despite the name, local memory is not a tiny fast store; it is backed by global memory and may only be cached opportunistically. Spills can turn a carefully fused kernel into a memory-bound one.

Memory Hierarchy

Memory hierarchy is the layered storage path between a processor and its data, ordered from large and slow to small and fast. On a GPU this usually means DRAM, L2 cache, shared memory, and registers, with performance determined by how often bytes cross each boundary.

The hierarchy exists because compute units can consume operands far faster than off-chip memory can deliver them. In inference, a token may need weights, KV cache entries, activations, and temporary fragments. If every use came directly from DRAM, tensor cores would often wait idle. The central performance question is not only how many operations a kernel performs, but how many bytes it must move through each memory level to perform them.

Concretely, data starts in DRAM, where model weights and KV pages live. Lines fetched from global memory pass through L2, which can serve repeated or neighbouring accesses across streaming multiprocessors. Shared memory is a per-SM scratchpad that kernels fill deliberately, often with tiles reused by many threads. Registers are private to each thread and hold live scalars, pointers, accumulator fragments, and operands close enough for the instruction pipeline to use directly.

The tradeoff is capacity and programmability. Registers are fastest but scarce; using too many can reduce occupancy or cause spills into memory. Shared memory is fast and cooperative, but only helps if the tile is reused enough and laid out without bank conflicts. Caches are convenient but unpredictable under irregular access. DRAM is large, but crossing to it dominates latency or bandwidth whenever arithmetic intensity is low.

Engineers meet the memory hierarchy when reading GPU kernel profiles, tuning attention, choosing batch sizes, or diagnosing why an inference server is slower than peak FLOP numbers suggest. FlashAttention, paged KV cache, quantisation, speculative decoding, and kernel fusion are all, in different ways, attempts to change where bytes travel, how often they are reused, or whether a costly DRAM read can be avoided.

Go deeper

Inference EngineeringThe Memory HierarchyTrace data through the GPU memory hierarchy: DRAM, L2 cache, shared memory, registers. Explain why each level exists and what it costs to cross.Open the lesson

Common questions

Is memory hierarchy just another name for cache?: No. Caches are part of the hierarchy, but the hierarchy also includes explicitly managed storage such as shared memory and registers. The distinction matters on GPUs because high-performance kernels often do not merely hope the cache works; they stage tiles, control reuse, and manage register pressure so data reaches compute units at the required rate.
Why does memory hierarchy matter so much for LLM inference?: Decode often has low arithmetic intensity: each new token may stream large weight matrices and read a growing KV cache while doing relatively little reuse. In that regime, tensor cores are not the limiting resource. Latency is set by bandwidth and locality, so arranging data movement can matter more than adding nominal compute capacity.
Does putting data in shared memory always make a kernel faster?: No. Shared memory helps when many threads reuse the staged data before it is evicted or overwritten. It can hurt if the tile is not reused, if accesses conflict across banks, or if the extra staging reduces occupancy. The honest answer depends on reuse, access pattern, register use, and whether memory latency was actually the bottleneck.
What are register spills and why are they bad?: A spill happens when a thread has more live values than can fit in physical registers, so some values are written to local memory. Despite the name, local memory is not a tiny fast store; it is backed by global memory and may only be cached opportunistically. Spills can turn a carefully fused kernel into a memory-bound one.

Every term