07.03 · Walkthrough
Cache-Aware Routing
Route on cache residency instead of load: send a request to the replica that already holds its prefix, and explain why round-robin, the correct answer for a stateless service, is the worst possible policy here.
Cache-aware routing sends an agent request to the replica that already owns the matching KV prefix, even when another replica looks less busy. For LLM serving, the prefix is GPU-resident state, not disposable text. Round-robin spreads that state across workers, causing repeated prefill, duplicated cache memory, and worse latency.
What this lesson answers
- why is round robin bad for LLM serving
- how does KV cache aware routing work
- when should routing prefer cache over load
Notes
Cache-aware routing is the serving policy that chooses an LLM replica by maximizing expected reuse of already-resident KV blocks for the request prefix, rather than minimizing queue length alone. For a request with tokenized prefix and replicas , let be the longest prefix length whose KV blocks are present and valid on replica ; the scheduler should minimize an estimated completion time such as , or equivalently choose when queues are comparable.
Common questions
- What is cache-aware routing for LLM replicas?
- Cache-aware routing chooses a replica based on whether it already has the request prefix in its KV cache. The scheduler estimates whether reusing cached prefix blocks beats sending the request to a shorter queue. For agent workloads with repeated prompts, tool schemas, and conversation history, that cache residency can dominate the routing decision.
- Why is round-robin routing harmful for agent serving?
- Round-robin is suitable when workers are stateless and interchangeable. LLM replicas with KV caches are not interchangeable once they have served repeated prefixes. Spraying similar requests across replicas forces each GPU to recompute and store the same prefix state, turning useful temporal locality into duplicated memory use and avoidable prefill work.
- When should a router ignore a KV cache hit?
- A cache hit is not always worth chasing. If the matching prefix is short, the warm replica has a much deeper queue, or memory pressure would evict more valuable prefixes, the scheduler should prefer another worker. Production policies usually combine queue delay with estimated prefill savings rather than using pure cache affinity.
Short definition: what is Cache-Aware Routing?