04.10 · Walkthrough
Throughput vs Latency
Read the batch-size curve: larger batches raise tokens per second and raise per-user latency at the same time. Pick a point on it deliberately instead of inheriting a default.
Batch size is a trade-off curve, not a tuning knob to maximise blindly. Larger batches usually improve total tokens per second by sharing expensive model work across requests, while also increasing inter-token latency, queueing, and tail risk. The right operating point is the knee that meets product latency targets.
What this lesson answers
- how does batch size affect inference latency
- why does batching improve tokens per second
- how to choose LLM serving batch size
Notes
Batching in autoregressive serving means evaluating one decode step for active sequences in a single forward pass, so the dominant weight read is amortized across emitted tokens. A useful first-order model for decode on one GPU is , where is model weight bytes read per step, is per-sequence KV traffic at context length , and is the matmul/attention work.
Common questions
- Why does a larger batch increase throughput but not make each user faster?
- A larger batch lets the server do one decode step for many active sequences, so expensive weight reads and kernel launches are shared. That raises aggregate tokens per second. Each individual request still waits for the whole batch step, plus any time spent waiting to join a batch, so user-visible latency can rise.
- Is tokens per second the same as per-user latency?
- No. Aggregate tokens per second measures total output across the device or service. Per-user latency is the time a single stream waits between tokens, including decode step time and queueing. A dashboard can show better throughput while the interactive experience becomes worse, especially under bursty traffic or long contexts.
- How should I pick a batch size for LLM serving?
- Measure throughput, time to first token, inter-token latency, and tail latency across realistic traffic. Choose the point where adding more concurrent sequences gives little extra throughput or starts violating latency goals. Do not inherit the framework default unless it was chosen for the same model, prompt mix, hardware, and service objective.
Short definition: what is Throughput vs Latency?