Why is batch size 1 so bad on a GPU?

Because the expensive model weights must be fetched from HBM to produce only one token’s activations. That turns the main linear layers into matrix-vector work with almost no weight reuse. GPUs are built to run large matrix operations with high arithmetic intensity, so this shape leaves compute units waiting on memory.

Does one request always mean batch size 1?

Not always. A long prompt during prefill contains many token positions, so even a single request can create a large matrix-shaped workload. The pathological case is decode, where each request usually contributes only its current next-token position. The important unit is active token rows at a layer, not the number of HTTP requests.

Why not always wait for a bigger batch?

Bigger decode batches improve weight reuse, but waiting increases queueing delay and can worsen tail latency. Past some point, the bottleneck may become KV-cache reads, communication, scheduler overhead, or logits processing. Production systems therefore use dynamic batching policies rather than a single fixed answer.

Do FlashAttention or paged KV caches solve this by themselves?

They solve related memory problems, but not the whole batch size 1 issue. FlashAttention reduces avoidable attention memory traffic, and paged KV caches make changing batches practical without copying long histories. The large projection and MLP weights still need multiple active token rows to be reused efficiently during decode.

One Request at a Time

One Request at a Time is the batch size 1 decode regime in LLM serving, where a GPU applies the model’s weight matrices to a single token position. It is usually the worst case for throughput because the weights are streamed from high-bandwidth memory with almost no reuse, making inference memory-bound rather than compute-bound.

The problem is that decoder inference generates tokens sequentially. For each next token, the model must run its projections and feed-forward layers again. If there is only one active token row, those layers look like matrix-vector products, not large matrix-matrix multiplies. The GPU still has to fetch the same enormous weight tensors from HBM, but it performs only the arithmetic needed for one token, so the tensor cores are starved.

Concretely, for a projection written as Y=XW, batching means X has many rows, each row representing a token position from some live request. The weight matrix W is read once and multiplied against all those rows before it is displaced from the cache hierarchy. With B=1, the reuse factor on W is one. Arithmetic intensity collapses to roughly a small constant per byte for FP16 weights, so bandwidth, not FLOP capacity, sets the pace.

The trade-off is latency and scheduling complexity. Waiting to form a larger batch can improve total tokens per second, but it can also delay an individual user’s next token or time to first token. The honest answer is workload-dependent: arrival rate, prompt length, sequence length, KV-cache traffic, all-reduce, logits processing, and service-level objectives determine whether more batching helps or merely moves the bottleneck somewhere else.

Engineers meet this in serving systems as continuous or in-flight batching. The scheduler rebuilds the active decode batch at each step because requests arrive, finish, and stop at different times. Systems such as vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, and llm-d use paged KV cache management and dynamic scheduling so unrelated requests can share each weight stream during decode. A common misunderstanding is that this is about API users; it is really about token rows per matrix operation.

Go deeper

Inference EngineeringOne Request at a TimeExplain why batch size 1 is the worst case for a GPU: the whole weight matrix is read from HBM to produce a single token, so arithmetic intensity collapses and the device runs at a few percent of peak.Open the lesson

Common questions

Why is batch size 1 so bad on a GPU?: Because the expensive model weights must be fetched from HBM to produce only one token’s activations. That turns the main linear layers into matrix-vector work with almost no weight reuse. GPUs are built to run large matrix operations with high arithmetic intensity, so this shape leaves compute units waiting on memory.
Does one request always mean batch size 1?: Not always. A long prompt during prefill contains many token positions, so even a single request can create a large matrix-shaped workload. The pathological case is decode, where each request usually contributes only its current next-token position. The important unit is active token rows at a layer, not the number of HTTP requests.
Why not always wait for a bigger batch?: Bigger decode batches improve weight reuse, but waiting increases queueing delay and can worsen tail latency. Past some point, the bottleneck may become KV-cache reads, communication, scheduler overhead, or logits processing. Production systems therefore use dynamic batching policies rather than a single fixed answer.
Do FlashAttention or paged KV caches solve this by themselves?: They solve related memory problems, but not the whole batch size 1 issue. FlashAttention reduces avoidable attention memory traffic, and paged KV caches make changing batches practical without copying long histories. The large projection and MLP weights still need multiple active token rows to be reused efficiently during decode.

Every term