Two Ways to Go Faster

Notes

In autoregressive decode, the mechanism to optimize is the per-token recurrence $x_{t+1}\sim p_{\theta }(\cdot \mid x_{\le t})$: every emitted token normally requires one forward pass that rereads model weights and attends over the accumulated KV cache. The useful framing is therefore $T_{\text{decode}}\approx N_{\text{steps}}\cdot T_{\text{step}}$, with $T_{\text{step}}\approx \max (B_{\text{read}}/\mathrm{BW},\mathrm{FLOPs}/FLOP/s)$; for large-batch prefill the max often selects compute, but for single-token decode of large models it usually selects bandwidth. The rest of the module hangs on the dichotomy: an optimization either lowers $B_{\text{read}}$ per accepted token, or lowers $N_{\text{steps}}$ per accepted token. Quantization, tensor parallel layout, KV-cache compression, MQA/GQA, FlashAttention-style IO-aware kernels, and PagedAttention-style cache management are “fewer bytes.” Speculative decoding, Medusa-style heads, and EAGLE are “fewer steps,” because they try to accept multiple future tokens after one expensive target-model verification.

A concrete bandwidth calculation makes the framing less slogan-like. A 70B parameter model in FP16 has about $70\times 10^9\cdot 2=140$ GB of weights, ignoring small metadata and non-parameter reads. On an 80GB H100 with peak HBM bandwidth around $3.35$ TB/s, a single decode step that must stream all weights has a lower bound $140/3350=0.0418$ s, or about $23.9$ tokens/s before KV traffic, synchronization, kernel overhead, and imperfect bandwidth utilization. KV reads are smaller but not free: with $L=80$ layers, $n_{kv}=8$ KV heads, head dimension $d=128$, FP16 elements, and context $S=8192$, reading K and V for one new token costs $2\cdot L\cdot n_{kv}\cdot d\cdot S\cdot 2=2,684,354,560$ bytes, about $2.68$ GB, or another $0.80$ ms at peak HBM. That is why shrinking weights from FP16 to INT4 is a first-order decode optimization, while shaving a few TFLOPs from an MLP is not first-order unless batch size or arithmetic intensity moves the step out of the bandwidth regime.

“Fewer bytes” techniques attack different terms in $B_{\text{read}}$. Weight-only quantization reduces the weight stream roughly from $2P$ bytes to $bP/8$ bytes plus scales; for the same 70B model, ideal 4-bit weights are $35$ GB, giving a bandwidth lower bound $35/3350=10.4$ ms, or $96$ tokens/s before overhead. Grouped-query attention, introduced as a practical compromise after multi-query attention, reduces KV cache bytes by replacing many query heads with fewer KV heads; the example above uses 8 KV heads rather than, say, 64, an $8\times$ reduction in the attention-cache stream. FlashAttention, from Dao et al.’s IO-aware exact attention paper and later FlashAttention-2/3, is a fewer-bytes optimization for prefill and attention kernels because it avoids materializing the $S\times S$ attention matrix and improves SRAM/HBM traffic. PagedAttention, introduced by vLLM in the SOSP 2023 paper “Efficient Memory Management for Large Language Model Serving with PagedAttention,” is also fewer bytes in practice: it prevents cache fragmentation and copying by mapping logical token blocks to physical pages, increasing effective batching without moving KV tensors around.

“Fewer steps” techniques change $N_{\text{steps}}$ rather than the per-step byte count. In speculative decoding, from Leviathan et al. and Chen et al. in 2023, a cheap draft model proposes $k$ tokens, the expensive target model verifies them in one parallel pass, and the sampler accepts a prefix with the exact target distribution; if the expected accepted tokens per target call is $a$, the target-model step count falls from $m$ to about $m/a$. If a 70B target at the FP16 lower bound above costs $41.8$ ms per call and a 7B draft costs roughly $14$ GB$/3.35$ TB/s $=4.2$ ms per generated draft token, then drafting $k=4$ tokens costs about $16.8$ ms and one target verification costs $41.8$ ms. With expected accepted length $a=3$, the cost per output token is $(41.8+16.8)/3=19.5$ ms, or $51$ tokens/s, versus $23.9$ tokens/s for target-only. EAGLE, from the 2024 “Extrapolation Algorithm for Greater Language-model Efficiency” line of work, improves the draft side by predicting in feature space rather than just token space, so its classification is still fewer steps: it is valuable only insofar as it raises $a$ or lowers draft cost.

The two classes fail in different, concrete regimes. Fewer-bytes methods stop helping when the bottleneck stops being HBM reads: large batch decode can raise arithmetic intensity enough that GEMMs become compute- or scheduler-limited, and then INT4 weights may be constrained by dequantization, tensor-core shape support, or collective communication rather than raw bytes. Overcompressing also makes quality or acceptance worse: aggressive KV quantization can perturb attention logits on long contexts, and a 4-bit target may reject more speculative tokens if it changes the distribution used by the serving contract. Fewer-steps methods become negative when $(C_{\text{target}}+C_{\text{draft}})/a>C_{\text{target}}$, equivalently $a<1+C_{\text{draft}}/C_{\text{target}}$. In the numeric example, $C_{\text{draft}}/C_{\text{target}}=16.8/41.8=0.40$, so speculative decoding must accept more than $1.40$ tokens per target verification merely to break even; on high-temperature sampling, code with brittle next-token dependencies, tool-call JSON, or a draft model poorly matched to the target, $a$ can fall below that and throughput drops.

The named inference systems are mostly combinations of this taxonomy rather than exceptions to it. vLLM’s PagedAttention is the canonical cache-management fewer-bytes system, while SGLang builds serving-level speedups around RadixAttention prefix reuse, constrained decoding, and speculative paths, so a proposed SGLang optimization should be classified by whether it avoids rereading KV/prefix bytes or avoids target decode steps. TensorRT-LLM packages fewer-bytes kernels and formats such as weight-only quantization, FP8, paged KV cache and fused attention, and also implements speculative decoding and in-flight batching. NVIDIA Dynamo, announced as a disaggregated generative-AI inference serving stack, and llm-d, the Kubernetes-native distributed LLM inference project, move the same primitives into cluster scheduling: prefix/KV routing and cache-aware placement are fewer bytes over the network and HBM, while speculative workers or disaggregated draft/verify pipelines are fewer steps. When evaluating any new paper, kernel, or release note, the operational question is not whether it “accelerates LLMs,” but which side of $N_{\text{steps}}\cdot B_{\text{read}}/\mathrm{BW}$ it reduces, and under what workload that product is still the limiter.