How is continuous batching different from normal batching?

Normal request-level batching forms a group, runs it until all members finish, and only then admits the next group. Continuous batching reforms the group after every generated token. Finished sequences leave immediately, and queued requests can enter the freed slots, so batch capacity follows the live workload rather than the longest request in an old batch.

Why does continuous batching often improve LLM throughput so much?

Decode is commonly limited by repeatedly moving model weights and KV data rather than by complex per-token computation. If a batch lane is empty, the server still pays much of the cost of the decode step but produces fewer tokens. Continuous batching keeps more lanes active under variable output lengths, converting otherwise idle capacity into useful generated tokens.

Does continuous batching always reduce latency?

No. It depends on traffic, context lengths, model shape, scheduler policy and service targets. If the queue is usually empty, there may be nothing to refill. If the scheduler waits too long to build bigger batches, individual users can see worse latency. Production systems usually cap waiting time, active tokens and prefill work to balance throughput against latency.

Is continuous batching the same as PagedAttention or FlashAttention?

No. Continuous batching is the decision to admit and remove requests at each decode step. PagedAttention is a KV-cache layout technique that makes frequent admission and retirement practical without expensive compaction. FlashAttention is an attention-kernel optimisation. They often appear together, but they solve different parts of the serving problem.

Continuous Batching

Continuous batching is an LLM serving policy that reschedules the active batch after every decode step, adding newly arrived requests and removing completed ones as tokens are generated. Instead of sealing a batch until its longest request finishes, it keeps GPU work slots occupied across uneven output lengths.

The problem is that autoregressive generation finishes at different times for different users. With a traditional fixed batch, every request enters together and the whole batch stays allocated until the slowest sequence is done. Short requests leave holes: the GPU still performs decode iterations, but fewer lanes produce useful tokens. Because decode repeatedly reads model weights and emits only one token per active sequence, those empty lanes translate directly into wasted memory bandwidth.

Continuous batching moves the scheduling boundary from the request to the token. After each forward pass, the server appends sampled tokens, checks which sequences reached an end condition or length cap, frees their sequence slots and KV-cache blocks, then admits waiting work into the available capacity. The next decode step is run for the newly formed active set. In practice, prefill and decode may be handled with separate rules, but the core idea is per-step retirement and admission.

The trade-off is that batching is not free. The scheduler and KV allocator now run constantly, and the system must manage fragmented, changing cache state. Larger active sets can also increase latency, especially if the server delays steps to gather more work. At long context lengths, KV-cache traffic may dominate the step, so adding sequences can hit memory bandwidth or capacity limits. Continuous batching improves occupancy; it does not make infinite batching sensible.

Engineers usually meet continuous batching in LLM serving runtimes under names such as in-flight batching or iteration-level scheduling. It is implemented by systems such as vLLM, SGLang and TensorRT-LLM, often alongside paged KV-cache management and attention-kernel optimisations. A common misunderstanding is to treat it as a model optimisation. It is a serving-loop policy: the transformer math is mostly unchanged, but the server stops letting completed requests leave idle batch lanes behind.

Go deeper

Inference EngineeringContinuous BatchingExplain iteration-level scheduling: admitting arriving requests and retiring finished ones at every decode step rather than every request, and why that alone is the largest single throughput win in LLM serving.Open the lesson

Common questions

How is continuous batching different from normal batching?: Normal request-level batching forms a group, runs it until all members finish, and only then admits the next group. Continuous batching reforms the group after every generated token. Finished sequences leave immediately, and queued requests can enter the freed slots, so batch capacity follows the live workload rather than the longest request in an old batch.
Why does continuous batching often improve LLM throughput so much?: Decode is commonly limited by repeatedly moving model weights and KV data rather than by complex per-token computation. If a batch lane is empty, the server still pays much of the cost of the decode step but produces fewer tokens. Continuous batching keeps more lanes active under variable output lengths, converting otherwise idle capacity into useful generated tokens.
Does continuous batching always reduce latency?: No. It depends on traffic, context lengths, model shape, scheduler policy and service targets. If the queue is usually empty, there may be nothing to refill. If the scheduler waits too long to build bigger batches, individual users can see worse latency. Production systems usually cap waiting time, active tokens and prefill work to balance throughput against latency.
Is continuous batching the same as PagedAttention or FlashAttention?: No. Continuous batching is the decision to admit and remove requests at each decode step. PagedAttention is a KV-cache layout technique that makes frequent admission and retirement practical without expensive compaction. FlashAttention is an attention-kernel optimisation. They often appear together, but they solve different parts of the serving problem.

Every term