What does the scheduler do in an LLM serving system?

The scheduler decides which waiting and active sequences should run in the next model step. It must fit work into compute and KV cache limits, mix prefill and decode work, respect queue policy, and avoid letting large or long-running requests monopolise the server.

Why is the KV cache manager a separate component?

KV memory is often the limiting resource during generation. The cache manager allocates, tracks, reuses, and frees the stored attention state for each sequence. Without careful cache management, the server may have idle compute but still be unable to admit more requests.

Where do REST and gRPC fit into model serving?

REST and gRPC usually sit at the frontend. They accept client requests, validate parameters, handle tokenisation or token IDs, and stream generated tokens back. They are not the generation engine itself, but poor frontend implementation can still add latency through blocking tokenisation or serialisation.

The Generation Loop

01.10 · Concept

The Serving Stack

Map the components of a production serving system: model runner, scheduler, KV cache manager, request queue, and REST/gRPC frontend.

A production LLM server is a token generation loop wrapped in admission control: API frontend, request queue, scheduler, model runner, and KV cache manager. The key design problem is choosing which requests run next while balancing GPU utilisation, latency, memory pressure, fairness, and streaming output.

What this lesson answers

LLM serving stack components in production
how does KV cache scheduling work
prefill versus decode in LLM serving

Notes

A production LLM serving stack is the set of components that repeatedly maps queued requests to batched decode steps: frontend $\to$ request queue $\to$ scheduler $\to$ model runner $\to$ KV cache manager $\to$ token output. Algorithm: receive request $r_{i} = (p_{i}, max_tokens_{i}, sampling_{i})$ , tokenize prompt $p_{i}$ , enqueue $r_{i}$ , select active batch $B_{t} = schedule (Q, C_{KV}, C_{tokens})$ , run model $y_{t}, K V_{t} = f_{θ} (x_{t}, K V_{< t})$ , sample next token $x_{t + 1} \sim P_{θ} (\cdot ∣ x_{\leq t})$ , update KV cache, stream…

References

Common questions

What does the scheduler do in an LLM serving system?: The scheduler decides which waiting and active sequences should run in the next model step. It must fit work into compute and KV cache limits, mix prefill and decode work, respect queue policy, and avoid letting large or long-running requests monopolise the server.
Why is the KV cache manager a separate component?: KV memory is often the limiting resource during generation. The cache manager allocates, tracks, reuses, and frees the stored attention state for each sequence. Without careful cache management, the server may have idle compute but still be unable to admit more requests.
Where do REST and gRPC fit into model serving?: REST and gRPC usually sit at the frontend. They accept client requests, validate parameters, handle tokenisation or token IDs, and stream generated tokens back. They are not the generation engine itself, but poor frontend implementation can still add latency through blocking tokenisation or serialisation.

Short definition: what is Serving Stack?

01.09 · The Generation LoopFirst Batch 01.11 · The Generation LoopYour Baseline