Is session affinity just sticky sessions for LLMs?

It is related, but the reason is different from ordinary web stickiness. The goal is not only to find in-memory application state, but to keep reusable transformer KV and prefix-cache blocks physically close to the request that will decode next. A session can be moved if the destination already has, or can cheaply receive, the relevant prefix state.

Why can a cache hit be unavailable even when the cluster has the data?

Prefix cache entries are useful only where they avoid prefill on the GPU doing the work. If turn one warmed worker A and turn two is sent to worker B, worker B may have no local KV blocks for the repeated prefix. A directory saying the prefix exists somewhere does not by itself remove the compute cost.

When should a router break session affinity?

Break it when the sticky worker is too queued, the session’s cached blocks have been evicted, the user is unlikely to return soon, or another placement has a better expected latency. The decision depends on expected hit probability, prefill time saved, KV memory pressure, and the extra queueing delay caused by keeping the session on one replica.

Session Affinity

Session affinity is a serving policy that keeps the turns of an interactive model session on the same worker, or on a worker that already has its reusable prefix state. Instead of balancing each request independently, the router uses session identity and cache locality to avoid recomputing shared prompt prefixes.

The need appears when an agent conversation is served as separate HTTP requests but the model input is cumulative. A later turn often resends the system prompt, earlier chat, documents, tool results, and the new user message. If that turn is routed to a replica that never processed the earlier context, the cluster may contain useful KV cache blocks somewhere, but the chosen GPU cannot use them locally.

Concretely, the router attaches a stable session key to requests and prefers a decode replica whose prefix cache already contains matching token blocks for that session. The engine canonicalises the input tokens, looks up block hashes or prefix-tree matches, and reuses cached KV for the unchanged prefix. Only the new suffix needs prefill work before decoding continues. The useful metric is per-session prefix-cache hit rate across turns, not just aggregate request throughput.

The trade-off is that stickiness spends memory and scheduling freedom. Cached prefixes occupy KV capacity, and keeping a session near one replica can increase queueing if that replica is busy while another is idle. Affinity also stops helping after eviction, after long gaps between turns, or when the expected prefill saving is smaller than the extra wait. The honest policy is conditional: keep affinity only while locality is likely and valuable.

Engineers meet session affinity in LLM gateways, inference schedulers, and Kubernetes-style serving stacks. In vLLM it relates to PagedAttention blocks and prefix caching; in SGLang to radix-tree prefix reuse; in TensorRT-LLM, Dynamo, and llm-d to KV reuse and placement. A practical test logs worker ID, input tokens, matched prefix tokens, prefill time, decode time, and KV allocation for the same trajectory with and without sticky routing.

Go deeper

Inference EngineeringSession AffinityTreat a session rather than a request as the unit of scheduling: measure prefix-cache hit rate across the turns of one real trajectory, and show what it costs when the second turn lands on a replica that has never seen the first.Open the lesson

Common questions

Is session affinity just sticky sessions for LLMs?: It is related, but the reason is different from ordinary web stickiness. The goal is not only to find in-memory application state, but to keep reusable transformer KV and prefix-cache blocks physically close to the request that will decode next. A session can be moved if the destination already has, or can cheaply receive, the relevant prefix state.
Why can a cache hit be unavailable even when the cluster has the data?: Prefix cache entries are useful only where they avoid prefill on the GPU doing the work. If turn one warmed worker A and turn two is sent to worker B, worker B may have no local KV blocks for the repeated prefix. A directory saying the prefix exists somewhere does not by itself remove the compute cost.
When should a router break session affinity?: Break it when the sticky worker is too queued, the session’s cached blocks have been evicted, the user is unlikely to return soon, or another placement has a better expected latency. The decision depends on expected hit probability, prefill time saved, KV memory pressure, and the extra queueing delay caused by keeping the session on one replica.

Every term