Is the serving stack the same thing as an inference server?

Not quite. An inference server may package the whole path, but the serving stack is the set of responsibilities inside it: admission, queueing, scheduling, model execution, KV cache management, sampling, and streaming. The distinction matters because bottlenecks often sit in the scheduler or cache allocator, not in the HTTP or gRPC layer.

Why does the scheduler matter so much for LLM serving?

The scheduler decides which requests share each prefill or decode step while respecting memory and token budgets. That choice controls accelerator utilisation, waiting time, fairness, and when new requests can be admitted. The best policy depends on workload shape: prompt lengths, generation lengths, service objectives, priorities, and available KV cache capacity.

What role does the KV cache manager play?

It allocates, tracks, and releases the stored key and value tensors used by attention for active sequences. Without it, every decode step would need to recompute earlier context. The cache manager is also a capacity gate: if there is not enough cache space for a sequence, the scheduler may have to delay, evict, or avoid admitting work.

Serving Stack

A serving stack is the production path that accepts model requests, queues them, schedules batched generation work, runs the model, manages KV cache memory, and streams tokens back. For LLMs, it is not a single server process but a coordinated loop around prefill, decode, batching, memory allocation, and request completion.

The serving stack exists because an LLM request is not a normal request-response computation. Each prompt first needs a prefill pass, then many small decode steps, and every active sequence carries growing KV cache state. If each user monopolised the accelerator until finished, utilisation would be poor and latency would vary wildly. Production serving is therefore mostly about interleaving unfinished generations without losing track of memory, fairness, or streaming outputs.

Concretely, a frontend receives REST or gRPC traffic, checks parameters, tokenises or accepts token IDs, and puts work into a request queue. A scheduler repeatedly chooses which waiting and running sequences form the next batch, subject to token and KV cache capacity. The model runner executes the transformer step, produces logits, samples the next token, and the KV cache manager records or frees the attention state needed by future decode steps.

The trade-off is that batching and continuous admission improve accelerator utilisation but can increase individual request latency and make behaviour less predictable. Prefill tends to be compute-heavy, while decode often stresses memory bandwidth because attention reads cached positions. KV cache is commonly the binding resource, not raw arithmetic. Scheduling policy also matters: FIFO, priority, deadlines, and fairness all optimise different notions of good service.

Engineers meet the serving stack when debugging slow streams, out-of-memory errors, poor batching, or uneven tail latency. In systems such as vLLM, HuggingFace TGI, and TensorRT-LLM, the names differ but the shape is recognisable: API handler or router, queue, scheduler, model execution layer, cache allocator, and streaming response path. A common misunderstanding is to treat the API server as the serving system; it is usually only the entrance.

Go deeper

Inference EngineeringThe Serving StackMap the components of a production serving system: model runner, scheduler, KV cache manager, request queue, and REST/gRPC frontend.Open the lesson

Common questions

Is the serving stack the same thing as an inference server?: Not quite. An inference server may package the whole path, but the serving stack is the set of responsibilities inside it: admission, queueing, scheduling, model execution, KV cache management, sampling, and streaming. The distinction matters because bottlenecks often sit in the scheduler or cache allocator, not in the HTTP or gRPC layer.
Why does the scheduler matter so much for LLM serving?: The scheduler decides which requests share each prefill or decode step while respecting memory and token budgets. That choice controls accelerator utilisation, waiting time, fairness, and when new requests can be admitted. The best policy depends on workload shape: prompt lengths, generation lengths, service objectives, priorities, and available KV cache capacity.
What role does the KV cache manager play?: It allocates, tracks, and releases the stored key and value tensors used by attention for active sequences. Without it, every decode step would need to recompute earlier context. The cache manager is also a capacity gate: if there is not enough cache space for a sequence, the scheduler may have to delay, evict, or avoid admitting work.

Every term