Is the scheduler loop just dynamic batching?

No. Dynamic batching says requests can be combined at runtime. A scheduler loop is the repeated mechanism that decides exactly which prompt chunks and decode steps enter each forward pass, while checking KV-cache pages, priorities, and preemption. In LLM serving, memory admission is as central as batch size.

Why are running decodes often prioritised over new prefills?

A running decode commonly needs only one more token of KV allocation, while a prefill may need a much larger chunk. Delaying a decode also directly increases inter-token latency for a user who is already receiving output. This is not a universal rule, since prompt-heavy workloads may justify more prefill capacity.

What happens when the KV cache is full?

The scheduler can stop admitting new work, choose smaller prompt chunks, or preempt existing work. Preemption usually means either moving KV pages to another memory tier or freeing them and later replaying the prompt and generated tokens to rebuild the cache. Which is better depends on interconnect bandwidth, model cost, and latency targets.

Does PagedAttention solve the scheduling problem?

PagedAttention makes scheduling more practical by allocating KV cache in blocks, so the engine can admit and evict at page granularity instead of needing contiguous reservations. It reduces waste and fragmentation, but the scheduler still has to choose winners when memory, bandwidth, or latency budgets are tight.

Scheduler Loop

A scheduler loop is the repeated admission-control cycle in an LLM serving engine that decides which waiting prompts and active decodes become the next GPU forward pass. It balances queues, token work, priorities, and KV-cache pages, admitting, delaying, preempting, swapping, or recomputing requests when memory is scarce.

The need for a scheduler loop comes from the fact that inference requests are not uniform jobs. Some are new prompts needing prefill over many tokens, while others are mid-generation and need only the next token. They all compete for GPU time and, more importantly, KV-cache memory. A serving engine cannot simply form a batch once and run it to completion, because every generated token changes the memory and latency constraints.

Concretely, each iteration starts with requests that are waiting and requests that are already running. Finished requests release their KV blocks. Active decodes are usually considered early because each needs a small new allocation and delay directly affects inter-token latency. The scheduler then admits prompt work, often in chunks, until it reaches limits such as batched tokens, active sequences, or available KV blocks. A block manager reserves cache pages rather than one large contiguous region.

The trade-off is that better utilisation creates more policy complexity and more failure modes. Chunking prefills can improve fairness, but it may delay first tokens or add overhead. Preemption avoids total stalls, but swap preemption burns host-device bandwidth and recompute preemption repeats model work. Paged KV allocation reduces fragmentation, but it does not invent memory. If most contexts are already near the cache limit, the loop may spend too much effort evicting and rebuilding state.

Engineers meet the scheduler loop in systems such as vLLM, SGLang, TensorRT-LLM, Dynamo, and llm-d, usually through knobs for maximum batched tokens, maximum sequences, chunked prefill, priority policy, prefix caching, and preemption mode. It also appears in production symptoms: rising time-to-first-token, uneven inter-token latency, low GPU utilisation, or sudden recomputation spikes. The honest tuning answer is usually “it depends” on prompt length, decode length, cache reuse, memory bandwidth, and service-level goals.

Go deeper

Inference EngineeringThe Scheduler LoopTrace what a serving engine does on one step: the waiting and running queues, admission against the KV budget, preemption and recomputation when memory runs short.Open the lesson

Common questions

Is the scheduler loop just dynamic batching?: No. Dynamic batching says requests can be combined at runtime. A scheduler loop is the repeated mechanism that decides exactly which prompt chunks and decode steps enter each forward pass, while checking KV-cache pages, priorities, and preemption. In LLM serving, memory admission is as central as batch size.
Why are running decodes often prioritised over new prefills?: A running decode commonly needs only one more token of KV allocation, while a prefill may need a much larger chunk. Delaying a decode also directly increases inter-token latency for a user who is already receiving output. This is not a universal rule, since prompt-heavy workloads may justify more prefill capacity.
What happens when the KV cache is full?: The scheduler can stop admitting new work, choose smaller prompt chunks, or preempt existing work. Preemption usually means either moving KV pages to another memory tier or freeing them and later replaying the prompt and generated tokens to rebuild the cache. Which is better depends on interconnect bandwidth, model cost, and latency targets.
Does PagedAttention solve the scheduling problem?: PagedAttention makes scheduling more practical by allocating KV cache in blocks, so the engine can admit and evict at page granularity instead of needing contiguous reservations. It reduces waste and fragmentation, but the scheduler still has to choose winners when memory, bandwidth, or latency budgets are tight.

Every term