04.03 · Walkthrough
Continuous Batching
Explain iteration-level scheduling: admitting arriving requests and retiring finished ones at every decode step rather than every request, and why that alone is the largest single throughput win in LLM serving.
Continuous batching keeps an LLM decode batch full by rescheduling after each generated token: finished requests leave, waiting requests enter, and the next forward pass runs on the current active set. It improves throughput because sealed batches lose capacity to short requests finishing early while long requests keep the whole batch open.
What this lesson answers
- what is continuous batching in LLM serving
- why does iteration scheduling improve LLM throughput
- continuous batching versus static batching decode
Notes
Continuous batching, more precisely iteration-level scheduling, is the serving policy that rebuilds the active decode batch after every generated token: at step , the scheduler forms , runs one forward pass for those requests, appends each sampled token, removes requests that hit EOS or length limits, and immediately admits waiting requests into the freed slots.
References
Common questions
- What is continuous batching?
- Continuous batching is a serving policy for autoregressive decoding where the active batch is updated after every token step. Requests that have completed are removed immediately, and queued requests can take their place. The batch is not fixed for the lifetime of the longest request, so the GPU spends more time doing useful token work.
- Why is continuous batching such a large throughput win?
- Decode repeatedly streams model weights and emits a small amount of new work per active sequence. If batch lanes sit idle after shorter generations finish, memory bandwidth is wasted. Continuous batching fills those lanes with new requests, raising the average active batch size without changing the model, kernels, quantisation, or parallelism strategy.
- When does continuous batching not help?
- It helps least when traffic is too low to keep a queue, when the model is already compute-bound at small batch sizes, or when long contexts make KV cache traffic the bottleneck. It can also hurt latency if the scheduler waits too long to gather more work instead of running the next decode step promptly.
Short definition: what is Continuous Batching?