What is the scheduler loop in an LLM serving engine?

It is the repeated decision that forms each GPU batch from waiting prompts and already running generations. Instead of building a fixed batch once, the engine revisits admission every step, accounting for newly required KV-cache blocks, token limits, sequence limits and request priority.

Why does KV cache matter so much for batching?

Every active request keeps keys and values for its previous tokens. Long contexts can consume most available memory even when only one new token is being generated. Page-based KV allocation reduces fragmentation and makes eviction practical, but it cannot make over-budget batches fit.

What happens when a running request must be preempted?

The engine can move its KV pages to host or storage memory, then restore them later, or discard them and replay the request to rebuild the cache. Swapping preserves compute but costs transfer bandwidth. Recomputation burns GPU work but can be better when transfer is slower than replay.

The Batch

04.04 · Walkthrough

The Scheduler Loop

Trace what a serving engine does on one step: the waiting and running queues, admission against the KV budget, preemption and recomputation when memory runs short.

A serving scheduler step turns queued prompts and active decodes into the next GPU forward pass by checking KV-cache capacity, token limits and priorities. It favours cheap decode progress, admits prefill chunks when budget allows, and may evict active work, either swapping KV pages out or freeing them for later recomputation.

What this lesson answers

how does an LLM serving scheduler work
why does KV cache limit batching
when should serving engines preempt requests

Notes

A serving scheduler step is the atomic decision that maps a set of waiting requests and partially decoded requests into one GPU forward pass while respecting the KV-cache budget. At step $t$ , the engine has a waiting queue $W$ , a running set $R$ , and a block allocator with free KV blocks $B_{free}$ ; it chooses a batch $S_{t} \subseteq W \cup R$ such that $\sum_{r \in S_{t}} Δ b_{r} \leq B_{free} + B_{reclaim}$ , where $Δ b_{r} = ⌈ Δ n_{r} / n_{tok/block} ⌉$ is the number of newly needed KV blocks.

References

vLLM: Optimization and Tuning

Common questions

What is the scheduler loop in an LLM serving engine?: It is the repeated decision that forms each GPU batch from waiting prompts and already running generations. Instead of building a fixed batch once, the engine revisits admission every step, accounting for newly required KV-cache blocks, token limits, sequence limits and request priority.
Why does KV cache matter so much for batching?: Every active request keeps keys and values for its previous tokens. Long contexts can consume most available memory even when only one new token is being generated. Page-based KV allocation reduces fragmentation and makes eviction practical, but it cannot make over-budget batches fit.
What happens when a running request must be preempted?: The engine can move its KV pages to host or storage memory, then restore them later, or discard them and replay the request to rebuild the cache. Swapping preserves compute but costs transfer bandwidth. Recomputation burns GPU work but can be better when transfer is slower than replay.

Short definition: what is Scheduler Loop?

04.03 · The BatchContinuous Batching 04.05 · The BatchChunked Prefill