Why is round-robin bad for LLM agent serving?

Round-robin ignores where the KV cache lives. If many requests share a system prompt or tool schema, it deliberately scatters them across replicas, making each worker recompute and store the same prefix. That may equalise arrivals, but it destroys temporal locality and turns reusable GPU-resident state back into repeated prefill work.

Is cache-aware routing just sticky sessions?

Not quite. Sticky sessions usually bind a user or session to a backend. Cache-aware routing binds a request to the replica with the best prefix state, which may be shared across many users and sessions. The routing key is not identity alone, but the token prefix and the validity of its cached KV blocks.

When should a router ignore the cache hit?

It should ignore or discount the hit when the matching prefix is short, the warm replica has a much deeper queue, the cache is likely to be evicted, or preserving that prefix would harm the global hit rate. Cache affinity is an input to scheduling, not an absolute rule. The right decision depends on prefill cost, queueing delay, and memory pressure.

Cache-Aware Routing

Cache-aware routing is a serving policy that sends an LLM request to the replica most likely to already hold valid KV cache blocks for its prompt prefix. Instead of treating replicas as interchangeable workers, it treats prefix state in GPU memory as part of the request, so cached context can be reused rather than recomputed.

The need appears because LLM serving is not stateless. Agent requests often repeat large prefixes: system instructions, tool definitions, formatting rules, retrieval scaffolding, and conversation history. A normal round-robin load balancer spreads those similar requests across replicas, which is reasonable for stateless HTTP but harmful here. It forces each GPU to rebuild and store the same prefix KV cache, wasting memory and prefill time that could have been avoided by returning to the warm replica.

Mechanically, the router needs a way to identify the request prefix and ask which worker owns matching cache blocks. Tokenised prompts are mapped to cache keys, often at block granularity, and each replica advertises or can be queried for its longest valid prefix match. The scheduler then compares cache affinity with queueing cost: a worker with a longer match may win even if it is slightly busier, because it can skip prefill for the cached prefix.

The trade-off is that locality can conflict with load balancing. A popular prefix can create a hotspot if every matching request is pinned to one replica, while a tiny cache hit may not justify waiting behind a long decode queue. Memory pressure also matters: preserving a large prefix can evict other useful prefixes. The honest policy is not pure affinity, but an estimated completion-time score that weighs queue depth, cache hit length, prefill cost, and eviction risk.

Engineers meet cache-aware routing in LLM serving stacks that expose prefix caching, paged KV storage, radix or block-based prefix indexes, and distributed schedulers. It shows up when serving agents, chat systems with shared templates, tool-heavy assistants, or multi-turn sessions. The key operational shift is that pods or replicas are no longer fungible endpoints. A request carries a dependency on GPU-resident prefix state, and routing must preserve that locality.

Go deeper

Inference EngineeringCache-Aware RoutingRoute on cache residency instead of load: send a request to the replica that already holds its prefix, and explain why round-robin, the correct answer for a stateless service, is the worst possible policy here.Open the lesson

Common questions

Why is round-robin bad for LLM agent serving?: Round-robin ignores where the KV cache lives. If many requests share a system prompt or tool schema, it deliberately scatters them across replicas, making each worker recompute and store the same prefix. That may equalise arrivals, but it destroys temporal locality and turns reusable GPU-resident state back into repeated prefill work.
Is cache-aware routing just sticky sessions?: Not quite. Sticky sessions usually bind a user or session to a backend. Cache-aware routing binds a request to the replica with the best prefix state, which may be shared across many users and sessions. The routing key is not identity alone, but the token prefix and the validity of its cached KV blocks.
When should a router ignore the cache hit?: It should ignore or discount the hit when the matching prefix is short, the warm replica has a much deeper queue, the cache is likely to be evicted, or preserving that prefix would harm the global hit rate. Cache affinity is an input to scheduling, not an absolute rule. The right decision depends on prefill cost, queueing delay, and memory pressure.

Every term