02.03 · Concept
The Memory Hierarchy
Trace data through the GPU memory hierarchy: DRAM, L2 cache, shared memory, registers. Explain why each level exists and what it costs to cross.
GPU inference speed is often decided by where bytes move, not by peak FLOPs. Weights and KV state start in HBM, pass through L2, may be staged in shared memory, and finally feed registers. Each boundary trades capacity for latency, bandwidth, programmer control and occupancy.
What this lesson answers
- why is LLM decode memory bandwidth bound
- how do GPU registers shared memory and L2 differ
- when does shared memory tiling help inference kernels
Notes
The GPU memory hierarchy is the set of progressively smaller, faster storage levels that a token’s computation crosses: HBM/DRAM holds model weights and KV pages, L2 caches recent global-memory lines across SMs, shared memory is an explicitly managed per-SM scratchpad, and registers hold each thread’s live scalars and fragments. The governing performance test is the roofline bound , where is FLOPs, is bytes crossing a given level, is compute throughput, and is bandwidth at that level.
Common questions
- Why does GPU memory hierarchy matter for inference latency?
- Inference repeatedly moves weights, activations and KV cache entries through storage levels with very different costs. If the kernel cannot reuse data before fetching more from HBM, latency follows memory bandwidth rather than tensor-core throughput. The practical question is which bytes cross which boundary, and how often.
- What is the difference between L2 cache and shared memory on a GPU?
- L2 is a hardware-managed cache shared across streaming multiprocessors, useful when nearby work touches the same global-memory lines. Shared memory is a per-SM scratchpad controlled by the kernel. It is faster and more predictable, but only helps when threads deliberately stage data that will be reused enough.
- Why can using more registers make a GPU kernel slower?
- Registers are the fastest place for live values, but they are limited. A kernel that uses too many registers per thread can reduce resident warps, leaving fewer independent instructions to hide memory stalls. If values spill, they may be written to local memory backed by HBM, which can erase the intended gain.
Short definition: what is Memory Hierarchy?