Static Batching

Notes

Static request-level batching means the server forms a fixed set of $B$ prompts, runs them through prefill together, and then advances the whole set through decode in lockstep until all requests in the set are complete. For a transformer using a dense attention kernel, the batched tensor is normally rectangular, so each prompt is padded to $L_{\max }={\max }_i{L}_i$ and the prefill attention work scales like ${\sum }_i{L}_{\max }^2$ rather than ${\sum }_i{L}_i^2$ if masking does not avoid the padded rows efficiently; even when FlashAttention avoids materializing the full attention matrix, the rectangular batch still drives scheduling and memory traffic around the maximum sequence length. The useful-token fraction is $u=\frac{{\sum }_{i=1}^B{L}_i}{B{L}_{\max }}$, and the padding waste is $1-u$. During decode, static batching also imposes a request-level barrier: if request $i$ finishes after $T_i$ generated tokens, its result is not returned or the batch slot not reused until $T_{\max }={\max }_i{T}_i$, creating head-of-line blocking of ${\sum }_i(T_{\max }-T_i)$ decode steps across completed requests.

On an 80GB H100 with about $3.35$ TB/s HBM bandwidth, take a 70B FP16 model whose weights are roughly $70\text{B}\times 2=140$ GB, so tensor parallel over two H100s leaves each GPU streaming about $70$ GB of weights per decode step. In the memory-bound decode regime, a crude roofline bound per GPU is $3.35\text{ TB/s}/70\text{ GB}\approx 47.9$ batch-steps/s; with batch $B=8$, that is at most $383$ token/s aggregate if KV reads and kernel overheads were free. Now suppose prompts have lengths $[256,512,768,1024,1024,1536,2048,4096]$. Static padding gives $B{L}_{\max }=8\times 4096=32768$ prompt token slots, while real tokens are $11264$, so $u=11264/32768=0.344$ and $65.6\%$ of the rectangular prefill slots are padding. If their output lengths are $[32,32,64,96,128,128,192,512]$, static decode runs $8\times 512=4096$ request-token steps but only $1184$ are useful; $2912$ slot-steps are pure head-of-line blocking.

The two costs show up differently in traces. Padding is paid mostly in prefill, where a short prompt beside a very long prompt inherits the launch geometry and masks of the long prompt; this is why production engines moved toward length bucketing even before full continuous batching. Head-of-line blocking is paid in decode, where generation is inherently one token at a time per live sequence: a request that emitted EOS at step $32$ can still occupy scheduler accounting, output buffering, and sometimes a KV-cache allocation until the request at step $512$ completes. Static batching was attractive in early GPT serving stacks because it maximized GEMM dimensions and made CUDA graph capture easy: the batch shape is known, the KV cache layout is simple, and there is no per-step admission control. But the mechanism is hostile to the actual distribution of chat traffic, where prompts and sampled completions are heavy-tailed and stop sequences, tool calls, and max-token limits produce highly variable $T_i$.

This optimization stops working when variance in either $L_i$ or $T_i$ dominates the throughput gain from larger matrix multiplications. A concrete rule of thumb is that if the useful-token fraction $u={\sum }_i{L}_i/(B{L}_{\max })$ falls below the speedup you obtained by moving from smaller to larger GEMMs, static prefill batching is already net negative. For example, if batching eight prompts makes prefill kernels $2.2\times$ faster than processing them separately but padding leaves $u=0.34$, the effective speedup is only $2.2\times 0.34=0.75$, a slowdown. In decode, the same logic applies to occupancy: if the mean completion is $148$ tokens but $T_{\max }=512$, a static batch keeps slots alive for $3.46\times$ the useful decode duration. Static batching is also poor for latency SLOs under bursty arrivals: a short request that joins a batch with one long completion sees its time-to-final-token coupled to the long request, even though the model could have freed the slot after EOS.

Modern LLM servers are largely reactions against these two static-batch costs. vLLM introduced PagedAttention in “Efficient Memory Management for Large Language Model Serving with PagedAttention” and implements continuous batching by admitting new requests into freed decode slots while representing KV cache as pageable blocks. SGLang’s runtime, described with RadixAttention and its serving system releases, combines continuous batching with prefix-cache-aware scheduling for agentic workloads. TensorRT-LLM added in-flight batching in NVIDIA’s release notes and examples, allowing requests to enter and leave an executing batch without waiting for a global batch boundary. NVIDIA Dynamo, announced as an inference serving framework for disaggregated and dynamic LLM serving, and llm-d, the Kubernetes-native LLM serving project, both assume dynamic admission rather than a static rectangular batch as the unit of work. FlashAttention, from Dao et al., reduces attention memory traffic but does not by itself remove request-level head-of-line blocking. EAGLE, the speculative decoding method, attacks decode-step count by proposing tokens, orthogonal to whether the serving scheduler is static or continuous.

The engineer’s mental model should therefore be precise: static batching is not “batching” in the abstract, but a fixed cohort contract. Its benefit is better accelerator utilization from larger, more regular kernels and fewer launches; its costs are exactly padding to the longest sequence in prefill and head-of-line blocking until the slowest request completes in decode. It is still defensible for homogeneous offline jobs, benchmark harnesses with equal prompt and output lengths, or embedding-style transformer passes where every item has the same shape. It is usually the wrong default for interactive LLM serving with mixed prompt lengths and stochastic generation. In that setting, the successor mechanisms preserve the useful part of batching, namely shared GPU execution, while breaking the fixed-cohort barrier that makes completed requests wait for the slowest sequence. Those mechanisms are vLLM’s PagedAttention scheduler, TensorRT-LLM in-flight batching, SGLang’s prefix-aware runtime, and related Dynamo or llm-d orchestration.