Two Ways to Go Faster
Two Ways to Go Faster is a framing for LLM inference optimisation: during autoregressive decode, speed usually improves by reading fewer bytes per generated token or by needing fewer target-model steps per generated token. It matters because decode is often limited by memory bandwidth, not by raw arithmetic throughput.
The problem is the decode loop. After prefill, an autoregressive model emits tokens one at a time: each new token depends on the prefix, then becomes part of the next prefix. For large models and small decode batches, the expensive part is commonly moving weights and KV-cache data through HBM again and again. So an optimisation that removes a little arithmetic may not matter if the chip is mostly waiting for bytes.
The useful mental model is that total decode time is the number of decode steps multiplied by the cost of each step. Each step rereads model weights and attends to stored keys and values from previous tokens. Fewer-bytes methods shrink or avoid those reads: quantised weights, smaller KV caches, grouped-query attention, IO-aware attention kernels, cache paging, and prefix reuse all reduce memory traffic or wasted movement.
Fewer-steps methods instead try to get more accepted output tokens from each expensive target-model call. Speculative decoding is the clearest example: a cheaper draft model proposes several future tokens, then the target model checks them in parallel and accepts a prefix without changing the target distribution. Medusa-style heads and EAGLE-like approaches are in the same class when their value comes from reducing target decode iterations.
The trade-off is that the classification does not guarantee a win. Fewer-bytes techniques can lose accuracy, add dequantisation overhead, complicate kernels, or stop helping once the workload becomes compute-, scheduler-, or communication-bound. Fewer-steps techniques depend on acceptance: if the draft is costly or poorly matched, the extra work can outweigh the saved target calls. The honest answer is workload-dependent.
Engineers meet this framing when reading inference release notes, choosing serving engines, or reviewing papers. vLLM-style paged KV management is fewer bytes. Weight-only quantisation is fewer bytes. FlashAttention-style kernels are fewer bytes, especially by reducing HBM traffic. Speculative decoding is fewer steps. When a vendor says an LLM stack is faster, ask which side changed: bytes per step, or target steps per accepted token.
Common questions
- Why is decode usually described as bandwidth-bound?
- In single-token decode, the model often performs a small amount of new computation while rereading a very large amount of state: weights and accumulated KV cache. The limiting resource is therefore often how fast bytes can be delivered from HBM, not how many FLOPs the accelerator can theoretically execute.
- Is quantisation a fewer-bytes or fewer-steps optimisation?
- Quantisation is normally a fewer-bytes optimisation. It stores weights, activations, or KV-cache entries in a smaller representation, so less data is read or moved during each decode step. It does not, by itself, reduce the number of autoregressive target-model calls needed to produce a sequence.
- Is speculative decoding always faster?
- No. Speculative decoding helps only when the accepted tokens per target verification are high enough to pay for the draft work and verification overhead. It can underperform on high-entropy sampling, mismatched draft models, brittle formats, or tasks where small token differences cause the draft to diverge quickly.
- Where do attention optimisations fit in this framing?
- It depends on what they change. Sliding-window or local attention can reduce how much KV history is read. Grouped-query or multi-query attention reduces KV-cache size. FlashAttention-style kernels reduce memory traffic by changing how attention is tiled and materialised. These are mainly fewer-bytes techniques, not fewer-steps techniques.