Why can throughput improve while latency gets worse?

Batching lets one forward pass serve multiple active sequences, so the hardware produces more total tokens for roughly the same model-weight read. But each user may wait to join a batch, then wait for a larger decode step, and may be delayed by prefill or scheduling work. The system is busier and more efficient, not necessarily more interactive.

Is the best batch size just the largest one that fits in memory?

No. The largest batch may maximise a benchmark-style throughput number while violating latency targets. Once KV traffic, compute, queueing, or paging grows, each added request can contribute little extra throughput and a lot of delay. The right point depends on context lengths, arrival pattern, model shape, hardware, scheduler, and the product’s latency objective.

What should I measure when choosing between throughput and latency?

Measure aggregate tokens per second together with first-token latency, inter-token latency, and tail latency under realistic traffic. Include mixed prompt lengths and bursty arrivals, not only steady synthetic requests. If speculative decoding, prefix caching, or disaggregated prefill is enabled, remeasure the curve, because those features shift the operating point rather than removing the trade-off.

Throughput vs Latency

Throughput is the total rate at which a serving system produces work, while latency is the time an individual request waits or takes to complete. In model inference, batching can improve tokens per second for the hardware while making each user wait longer for admission, decode steps, or prefill interruptions.

The distinction matters because production inference is usually shared. A GPU can look efficient on an aggregate tokens-per-second chart while a single user sees slow token streaming or a long wait for the first token. Batching creates this tension deliberately: it keeps expensive hardware busier, but it also couples independent requests together. The mistake is to treat a higher throughput number as if it automatically meant a faster experience for every caller.

In autoregressive serving, a decode step emits the next token for each active sequence in a batch. The model weights are largely read once for that step, so serving several sequences together amortises that dominant read across several emitted tokens. Throughput rises because more tokens come out per forward pass. Latency rises because the step itself may become heavier, requests may queue until admitted, and long prefills can interrupt decode work.

The trade-off bends as batch size, context length, KV-cache traffic, and compute pressure grow. At small batches the GPU may be underused, so adding requests is cheap. Later, attention over longer histories, memory bandwidth, scheduler overhead, and paging can dominate. A common misunderstanding is that per-user latency is just the inverse of aggregate throughput. It is not, because queueing and shared-step duration are visible to users.

Engineers meet this curve in serving frameworks that implement continuous or inflight batching, such as systems using PagedAttention, prefix reuse, or separate prefill and decode scheduling. The useful question is not whether batching is good, but where to operate on the curve. Pick limits for active sequences, batched tokens, and admission policy from measured first-token latency, inter-token latency, tail latency, and total throughput, not from a default benchmark setting.

Go deeper

Inference EngineeringThroughput vs LatencyRead the batch-size curve: larger batches raise tokens per second and raise per-user latency at the same time. Pick a point on it deliberately instead of inheriting a default.Open the lesson

Common questions

Why can throughput improve while latency gets worse?: Batching lets one forward pass serve multiple active sequences, so the hardware produces more total tokens for roughly the same model-weight read. But each user may wait to join a batch, then wait for a larger decode step, and may be delayed by prefill or scheduling work. The system is busier and more efficient, not necessarily more interactive.
Is the best batch size just the largest one that fits in memory?: No. The largest batch may maximise a benchmark-style throughput number while violating latency targets. Once KV traffic, compute, queueing, or paging grows, each added request can contribute little extra throughput and a lot of delay. The right point depends on context lengths, arrival pattern, model shape, hardware, scheduler, and the product’s latency objective.
What should I measure when choosing between throughput and latency?: Measure aggregate tokens per second together with first-token latency, inter-token latency, and tail latency under realistic traffic. Include mixed prompt lengths and bursty arrivals, not only steady synthetic requests. If speculative decoding, prefix caching, or disaggregated prefill is enabled, remeasure the curve, because those features shift the operating point rather than removing the trade-off.

Every term