One Request at a Time

Notes

Batching in decoder-only inference is the mechanism of applying the same layer weights to multiple independent token positions before those weights are evicted from the GPU memory hierarchy. For a linear projection $Y=XW$, with $X\in {\mathbb{R}}^{B\times d}$ and $W\in {\mathbb{R}}^{d\times d^{\prime }}$, the per-token batch is $B$ rows, and the arithmetic intensity is approximately $AI=\frac{2Bd{d}^{\prime }}{s(d{d}^{\prime }+Bd+B{d}^{\prime })}$ FLOP/byte, where $s$ is bytes per element and the dominant term at small $B$ is the weight read $sd{d}^{\prime }$. When $B=1$, this reduces to roughly $AI\approx 2/s$, or about $1$ FLOP/byte for FP16, because every parameter is streamed from HBM to create one token’s activations. The GPU is then not “doing one user quickly”; it is repeatedly launching enormous matrix-vector products whose reuse factor on $W$ is one. Continuous batching in Orca, later industrialized in vLLM’s PagedAttention paper and systems such as SGLang, TensorRT-LLM, NVIDIA Dynamo, and llm-d, is specifically about raising this reuse factor during decode.

A concrete H100 calculation makes the failure mode visible. Take a 70B-parameter decoder in FP16, so the weights are about $70\times 10^9\times 2=140$ GB. On an 80GB H100 with HBM3 bandwidth around $3.35$ TB/s, a single-token decode that must stream the full model has a lower-bound weight-read time of $140/3350=0.0418$ s, or about $23.9$ tokens/s, before KV-cache traffic, collectives, launch overheads, and non-GEMM work. The math for that token is about $2\times 70$B $=140$ GFLOP, so at $23.9$ tok/s the chip is doing roughly $3.35$ TFLOP/s, only about $1.7\%$ of a $1979$ TFLOP/s FP16 tensor-core peak H100. This is not because the model is too small for the GPU; it is because $AI\approx 140$ GFLOP$/140$ GB $=1$ FLOP/byte, while the H100 roofline knee is around $1979/3.35\approx 591$ FLOP/byte. Batch size 1 is hundreds of times too memory-bound to approach tensor-core peak.

Increasing decode batch changes the same calculation because the weight bytes are amortized across $B$ tokens. Ignoring activation and KV traffic, a batch $B=64$ decode of the same 70B model reads about the same $140$ GB of weights but performs $64\times 140$ GFLOP $=8.96$ TFLOP, so $AI\approx 64$ FLOP/byte. That is still below the H100 knee, but it is sixty-four times better than batch 1 and can move the realized throughput from single-digit TFLOP/s into the hundreds once kernels become large GEMMs rather than GEMVs. The limiting per-token weight-bandwidth cost becomes $140/64=2.19$ GB, implying an optimistic $3350/2.19\approx 1530$ tok/s if nothing else bottlenecks. This is the reason serving stacks do not reserve a GPU for one request unless latency isolation dominates economics: the exact same parameter stream can produce many next-token logits. FlashAttention helps the attention part by tiling QK/V through SRAM, but during decode the large MLP and projection weights still demand cross-request batching for reuse.

The implementation detail is that requests do not naturally stay aligned: one prompt is prefilled, another is decoding token 37, another stops on EOS. Orca’s “iteration-level scheduling” paper made the key observation that the batch should be rebuilt every decode step, not only at request boundaries. vLLM’s 2023 PagedAttention paper then made this practical by paging the KV cache so a changing logical batch can reference non-contiguous KV blocks without copying long sequences. SGLang extends this style with a runtime for structured generation and RadixAttention-style prefix reuse; TensorRT-LLM exposes in-flight batching and paged KV kernels in NVIDIA’s optimized engine; NVIDIA Dynamo and llm-d package disaggregated prefill/decode and distributed serving around the same need to keep decode batches dense. EAGLE, from the speculative decoding paper of that name, attacks a different term: it drafts multiple candidate tokens so one expensive target-model pass may accept several tokens, increasing useful work per weight stream when the verifier batch would otherwise be thin.

There are concrete regimes where batching stops helping or hurts. First, prefill for long prompts is already a matrix-matrix or large attention workload: a single request with $L=4096$ prompt tokens has $B_{\text{effective}}=4096$ positions through the weights, so adding unrelated prefills may only increase queueing delay or push attention into KV-memory pressure. Second, decode batching beyond the point where KV-cache reads, logits processing, all-reduce, or scheduler overhead dominate will not improve tokens/s linearly; with grouped-query attention using $8$ KV heads and head dimension $128$ in FP16, each layer reads roughly $2\times L\times 8\times 128\times 2=4096L$ bytes of K and V per decoded token. At $L=8192$ and $80$ layers, that is $4096\times 8192\times 80\approx 2.68$ GB/token of KV traffic, comparable to the $2.19$ GB/token amortized weight traffic at $B=64$. Third, batching worsens time-to-first-token and per-request tail latency when arrivals are sparse or service-level objectives are strict, because the scheduler waits or interleaves more work before a user’s next token.

Batch size 1 is therefore the pathological decode case for the GPU execution model: almost every layer degenerates to a GEMV-shaped operation, the HBM stream of model weights is consumed once, and tensor cores sit underfed. The important distinction is not “one request” versus “many users” at the API layer, but “one token row” versus “many token rows” at each weight matrix. PagedAttention, FlashAttention, and TensorRT-LLM kernels reduce avoidable memory movement inside attention and KV management; vLLM, SGLang, Dynamo, and llm-d keep the live token rows numerous enough that the same weights are reused; EAGLE manufactures additional verified progress per expensive model invocation. When the active decode set is small, the roofline arithmetic is unforgiving: $AI$ collapses toward $1$ FLOP/byte for FP16 weights, so an H100-class accelerator behaves like a very expensive memory streaming device rather than a near-petaflop matrix engine.