Why is round-robin bad for LLM serving?

Round-robin ignores where the useful KV cache lives. In a multi-turn session, the next prompt usually extends a prefix that one replica has already computed. Sending the continuation to another replica forces prefill again or misses the cached blocks. It may balance request counts while increasing actual GPU work and tail latency.

Is session affinity the same as KV-cache-aware routing?

Session affinity is the simple case: send the same conversation back to the replica that handled it before. KV-cache-aware routing is broader. It can route unrelated requests to a worker that already holds a shared prefix, such as a system prompt or tool schema, while still considering queue length and memory pressure.

When should a router ignore cache locality?

It should ignore or discount locality when the matching worker is overloaded, the reusable prefix is small, the session is stale, or traffic is mostly unique. In those cases, better batching and lower queueing can matter more than a cache hit. Production policies usually cap affinity, decay old prefix scores, and break stickiness under load.

Does the router move KV cache between replicas?

Usually the router only chooses a destination using metadata about where cache blocks are resident. Some disaggregated or advanced serving systems can transfer KV state between workers, but that has a cost. Whether transfer wins depends on prefix size, interconnect speed, expected future reuse, and current queueing pressure.

Routing Requests

Routing requests is the fleet-level policy that chooses which model replica should handle an inference request, using session affinity, prefix matches, queue state, and KV-cache residency instead of treating workers as identical. In LLM serving, good routing preserves expensive cached context; bad routing can turn every continuation into fresh prefill work.

The need for routing comes from the fact that replicas are not really interchangeable once they have served traffic. A worker may hold the KV cache for a long conversation, a shared system prompt, a tool schema, or a retrieval template. If the next related request is sent elsewhere, that cached prefix is useless. Plain round-robin is commonly misunderstood as neutral load balancing, but for LLMs it actively erases locality by scattering continuations across workers.

A practical router keeps metadata about cache ownership, not the KV tensors themselves. It records token-prefix hashes, block ranges, the replica that owns them, freshness, and sometimes session ownership. On each request it first checks whether the conversation is already attached to a healthy, non-saturated replica. If not, it compares candidate replicas by longest reusable prefix, then subtracts costs for queueing, memory pressure, and occupied decode capacity before choosing a worker.

The trade-off is that locality is not always worth chasing. A replica with the best prefix match may already have a long decode batch, little free KV memory, or poor batching compatibility. Hot prompts can overload one worker unless the system deliberately replicates the prefix or transfers KV state. Low-reuse traffic can make affinity harmful by adding router work, polluting cache, and preventing the serving engine from forming efficient batches.

Engineers meet request routing in inference gateways, Kubernetes-aware LLM serving stacks, and engines exposing prefix caching or paged KV-cache controls. The key implementation details are usually outside the model code: tokenizer-stable prefix hashing, session identifiers, health checks, queue metrics, cache eviction signals, and fallbacks when affinity is broken. The honest rule is not “always route to the old replica”, but price reuse against load, batching, memory, and transfer.

Go deeper

Inference EngineeringRouting RequestsRoute across replicas by KV-cache locality and session affinity, and explain why plain round-robin destroys the prefix hit rate the KV cache module worked to build.Open the lesson

Common questions

Why is round-robin bad for LLM serving?: Round-robin ignores where the useful KV cache lives. In a multi-turn session, the next prompt usually extends a prefix that one replica has already computed. Sending the continuation to another replica forces prefill again or misses the cached blocks. It may balance request counts while increasing actual GPU work and tail latency.
Is session affinity the same as KV-cache-aware routing?: Session affinity is the simple case: send the same conversation back to the replica that handled it before. KV-cache-aware routing is broader. It can route unrelated requests to a worker that already holds a shared prefix, such as a system prompt or tool schema, while still considering queue length and memory pressure.
When should a router ignore cache locality?: It should ignore or discount locality when the matching worker is overloaded, the reusable prefix is small, the session is stale, or traffic is mostly unique. In those cases, better batching and lower queueing can matter more than a cache hit. Production policies usually cap affinity, decay old prefix scores, and break stickiness under load.
Does the router move KV cache between replicas?: Usually the router only chooses a destination using metadata about where cache blocks are resident. Some disaggregated or advanced serving systems can transfer KV state between workers, but that has a cost. Whether transfer wins depends on prefix size, interconnect speed, expected future reuse, and current queueing pressure.

Every term