Why the GPU Waits

Notes

The hardware floor for LLM inference is the minimum time imposed by the slower of three resources: floating-point throughput, memory bandwidth, and launch/scheduling overhead. In roofline form the per-token lower bound is $t\ge \max (F/P_{\mathrm{peak}},B/W_{\mathrm{mem}},t_{\mathrm{overhead}})$, where $F$ is FLOPs issued for the token, $P_{\mathrm{peak}}$ is effective GPU compute rate, $B$ is bytes moved from HBM, and $W_{\mathrm{mem}}$ is HBM bandwidth. The GPU is a throughput machine: it wants thousands of independent lanes kept busy with long streams of arithmetic or memory transactions. The CPU is latency-optimized: it is good at branching, scheduling, and initiating small tasks, but it cannot feed the GPU one tiny matrix-vector operation at a time without leaving gaps. “Why the GPU waits” usually means one of three concrete stalls: Tensor Cores have no ready tiles, HBM transactions are the limiter, or the kernel stream is fragmented by CPU/framework overhead.

In prefill, a transformer looks like large matrix-matrix multiplication, so arithmetic intensity is high and the GPU can approach the compute roof. In decode for batch size 1, each new token is closer to matrix-vector multiplication over all weights, so the same weights are reread for very little reuse. A 70B dense model in FP16 has roughly $140$ GB of weights. On an 80GB H100 with $W_{\mathrm{mem}}=3.35$ TB/s, even ignoring KV reads and activations, one forward token cannot take less than $140/3350=0.0418$ s, or about $23.9$ tokens/s, if the weights must stream from HBM. The FLOP count is about $2N=140$ GFLOP per token for $N=70$B parameters; even at a conservative effective $500$ TFLOP/s this is $0.28$ ms, far below the $41.8$ ms bandwidth floor. That is the memory-bandwidth-bound regime: Tensor Cores are mostly waiting for weights, not multiplying.

Batching changes the arithmetic intensity because the same streamed weights serve multiple sequences. If batch $b=32$ decode tokens are multiplied by the same 70B weights, the traffic is still about $140$ GB for weights, but the useful arithmetic rises to $32\times 140$ GFLOP $=4.48$ TFLOP, raising intensity from roughly $1$ FLOP/byte to $32$ FLOP/byte. The bandwidth floor remains $41.8$ ms for the whole batch, which is $765$ tokens/s aggregate before KV and overhead; the compute time at $500$ TFLOP/s is $4.48/500=8.96$ ms, still below bandwidth. Around $b\approx W_{\mathrm{mem}}/P_{\mathrm{eff}}$ inverted through intensity, the bottleneck can move toward compute, and for very large prefill matrices FlashAttention from Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” and FlashAttention-2 reduce HBM traffic enough that attention kernels become compute-competitive rather than pure memory traffic.

The overhead-bound regime is different: the GPU waits because the host and runtime are feeding it kernels, copies, synchronization points, and request decisions too slowly. A single decode step may involve attention, MLP projections, elementwise ops, sampling, logits processing, cache address lookup, and scheduler bookkeeping; if these are many small kernels at low batch, tens of microseconds of launch and dependency latency can dominate useful work. vLLM’s PagedAttention, from Kwon et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention” and the vLLM 2023 release, attacks this by paging KV cache blocks so continuous batching can keep more sequences resident without huge contiguous allocations. SGLang’s RadixAttention and runtime, TensorRT-LLM’s inflight batching and fused kernels, NVIDIA Dynamo’s disaggregated serving stack, and llm-d’s Kubernetes-native serving architecture all exist because the hardware floor is not just a CUDA kernel; it includes request admission, prefix reuse, KV placement, and inter-GPU routing.

The KV cache makes the bandwidth floor token-position dependent. With grouped-query attention using $8$ KV heads and head dimension $128$ in FP16, each layer stores per token $2\times 8\times 128\times 2=4096$ bytes for keys and values. For an $80$-layer 70B-style model, that is $4096\times 80=327,680$ bytes, or about $320$ KiB per cached token per sequence. Attending over a $16,384$-token context therefore reads on the order of $327,680\times 16,384=5.37$ GB of KV data per generated token if implemented naively across layers. At $3.35$ TB/s that alone is $1.60$ ms, much less than the $41.8$ ms weight-streaming floor at batch 1 but material at high batch or with quantized weights. PagedAttention improves allocation and batching, FlashAttention improves IO within attention, and TensorRT-LLM-style paged KV plus fused attention reduce waste, but none abolishes the fact that longer context turns decode into a KV bandwidth problem.

The techniques stop working when they move the bottleneck instead of removing it. Continuous batching helps until latency SLOs are violated, the batch becomes compute-bound, or KV cache capacity forces eviction, recomputation, or cross-device traffic. PagedAttention can make things worse for short, uniform prompts because page-table indirection and non-contiguous block gathers add overhead without solving fragmentation. FlashAttention helps when attention IO dominates, but for batch-1 decode of a huge dense model the weight matrix traffic dominates, so an optimized attention kernel barely moves tokens/s. Speculative decoding with EAGLE, from Li et al. “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” can reduce target-model steps when draft acceptance is high, but it hurts when the draft model is poorly matched, sampling temperature lowers acceptance, or verification batches are too small to amortize extra kernels. Dynamo, llm-d, vLLM, SGLang, and TensorRT-LLM expose the same engineering truth: once compute, HBM, or overhead is saturated, another layer of scheduling only reshuffles the wait.