Is static batching the same as ordinary batching?

No. Static batching is a particular batching contract: a fixed set of requests enters together and remains tied together until the slowest request completes. Batching in general just means executing multiple items together. Continuous batching, for example, still batches GPU work but can admit new requests when old ones finish.

Why does padding hurt prefill so much?

Transformer prefill processes the prompt tokens and builds attention state. In a static rectangular batch, the longest prompt determines the effective sequence dimension for the group. Short prompts are padded and masked, but the kernels and memory schedule are still shaped around the longest member. The more uneven the prompt lengths, the less of the batch is useful work.

What is head-of-line blocking in static decode?

Decode generates one token step at a time for each sequence. In a static batch, a request that reaches an end condition early is still coupled to the batch’s longest-running request. Its response, slot reuse, scheduler accounting, or cache allocation may be delayed, so short generations inherit latency from long ones.

When is static batching still a reasonable choice?

It depends on variance. Static batching can work well when requests have similar prompt lengths and similar output lengths, such as controlled offline evaluation or fixed-shape processing. It becomes unattractive when the wasted padded prefill and blocked decode time outweigh the gain from larger, more regular GPU kernels.

Static Batching

Static batching is request-level batching where an inference server groups a fixed cohort of prompts, runs prefill for that cohort, then decodes every member in lockstep until the slowest request finishes. It improves regularity for GPU work, but pays through padding to the longest input and head-of-line blocking during generation.

The problem static batching tries to solve is that accelerators dislike tiny, irregular units of work. A single LLM request often cannot keep a GPU well occupied, especially during prefill and decode kernel launches. Grouping requests gives larger tensor operations, simpler scheduling, and more predictable memory layouts. The catch is that real traffic is not uniform: one user sends a short prompt, another sends a long document, and their generated outputs stop at different times.

Mechanically, the server forms a fixed group and treats it as the unit of execution. During prefill, prompts are packed into a rectangular batch, so shorter prompts are padded up to the longest prompt in that group and masked where necessary. During decode, each live sequence advances one token step at a time, but the group remains together. If one sequence ends early, the static batch still continues until the longest generation in the cohort is done.

The two main costs are easy to name but often underestimated. Padding wastes prefill work and memory movement because the batch shape is determined by the maximum prompt length, not the average. Head-of-line blocking wastes decode capacity and increases latency because completed requests cannot fully leave, or their slots cannot be reused, until the slowest member finishes. Better attention kernels may reduce some padding overhead, but they do not remove the fixed-cohort barrier.

Engineers meet static batching in simple PyTorch serving loops, early LLM serving stacks, benchmark harnesses, and offline jobs where inputs are deliberately shaped alike. It can still be sensible for homogeneous workloads, fixed-length evaluations, or transformer passes with uniform item sizes. It is usually a poor default for interactive chat serving, where prompt lengths, stop conditions, tool calls, and sampled output lengths vary widely. Continuous or in-flight batching keeps GPU sharing while allowing requests to enter and leave dynamically.

Go deeper

Inference EngineeringStatic BatchingDescribe request-level batching and name its two costs: padding to the longest sequence, and head-of-line blocking where every finished request waits for the slowest one.Open the lesson

Common questions

Is static batching the same as ordinary batching?: No. Static batching is a particular batching contract: a fixed set of requests enters together and remains tied together until the slowest request completes. Batching in general just means executing multiple items together. Continuous batching, for example, still batches GPU work but can admit new requests when old ones finish.
Why does padding hurt prefill so much?: Transformer prefill processes the prompt tokens and builds attention state. In a static rectangular batch, the longest prompt determines the effective sequence dimension for the group. Short prompts are padded and masked, but the kernels and memory schedule are still shaped around the longest member. The more uneven the prompt lengths, the less of the batch is useful work.
What is head-of-line blocking in static decode?: Decode generates one token step at a time for each sequence. In a static batch, a request that reaches an end condition early is still coupled to the batch’s longest-running request. Its response, slot reuse, scheduler accounting, or cache allocation may be delayed, so short generations inherit latency from long ones.
When is static batching still a reasonable choice?: It depends on variance. Static batching can work well when requests have similar prompt lengths and similar output lengths, such as controlled offline evaluation or fixed-shape processing. It becomes unattractive when the wasted padded prefill and blocked decode time outweigh the gain from larger, more regular GPU kernels.

Every term