How is RadixAttention different from ordinary KV caching?

Ordinary KV caching keeps K and V for an active sequence so later decode tokens do not recompute earlier tokens. RadixAttention adds cross-request reuse: it indexes cached prefixes by token sequence, finds the longest prefix shared with a new request, and attaches that request to the existing KV entries before computing only the unmatched suffix.

How is RadixAttention different from vLLM prefix caching?

RadixAttention uses a compressed tree over token sequences, so it can match and split at token-level prefix boundaries. vLLM’s prefix caching uses hashed KV blocks, so reuse happens through consecutive matching blocks. The tree gives direct longest-prefix semantics; the hash approach aligns naturally with fixed pages, immutable block keys, and high-throughput block management.

When does RadixAttention not help?

It helps least when requests do not share early tokens, when shared text is shorter than the useful cache granularity, or when KV memory is already scarce. A unique user preamble, timestamp, nonce, or retrieved passage near the start can make the longest match tiny. Then the system pays tree and allocator overhead without avoiding much prefill.

Does RadixAttention reduce the memory used by the KV cache?

Not inherently. It can reduce duplicate computation by reusing existing KV tensors, but those tensors still occupy memory while cached. In some workloads it may increase pressure because long prefixes are kept in hope of reuse. The benefit depends on whether saved prefill work outweighs metadata, fragmentation, eviction churn, and occupied KV capacity.

RadixAttention

RadixAttention is SGLang’s prefix-cache design for LLM inference, storing KV-cache references in a compressed radix tree keyed by token sequences. When a new request arrives, the runtime finds the longest cached token prefix, reuses its existing K and V tensors, and computes prefill only for the remaining suffix.

The problem is repeated prefill. Many production prompts share long beginnings: system messages, chat templates, tool descriptions, agent scaffolding, or program branches that start the same way. Without prefix reuse, each request streams the model over those same prompt tokens again and again. The KV cache already contains the result of that work, but the engine needs a precise way to decide whether the cached tensors correspond to this request’s token prefix.

RadixAttention represents cached token strings as paths in a compressed radix tree. An edge can stand for a span of tokens, not just a single token. Lookup walks the request tokens through the tree until the match stops, giving the longest reusable prefix. Insertion may split an edge when two sequences share only part of a span. Tree nodes hold metadata and point through a memory pool to the physical KV storage used by attention.

The tradeoff is that RadixAttention buys finer-grained prefix sharing with more mutable runtime state. The tree must handle splitting, reference counts, eviction, allocator bookkeeping, and concurrent requests. It also consumes KV memory for prefixes that may or may not be reused. If overlap is low, or if a unique timestamp, nonce, or retrieved passage appears before the stable text, the lookup adds overhead while saving little prefill compute.

Engineers usually meet RadixAttention when comparing SGLang with systems such as vLLM. SGLang’s approach treats token sequences like ordered strings and asks for the longest prefix in a radix tree. vLLM’s automatic prefix caching is hash-based and works at KV block granularity on top of paged attention machinery. Neither design is universally better; the right fit depends on prompt structure, cache pressure, batching, and how the serving stack manages KV pages.

A common misunderstanding is that RadixAttention is a new attention kernel like FlashAttention. It is not. It is a cache-indexing and scheduling mechanism around the KV cache. The attention computation still reads K and V for the matched prefix, but the expensive prefill that would have produced those tensors is skipped. Speculative decoding methods such as EAGLE are also separate: they reduce decode steps, while RadixAttention removes repeated prefix prefill.

Go deeper

Inference EngineeringRadixAttentionCompare SGLang's radix tree approach to vLLM's hash-based prefix caching. Trace the radix data structure through radix_cache.py, radix_attention.py, and memory_pool.py. Two designs, different bets --- understand the tradeoffs without choosing a winner.Open the lesson

Common questions

How is RadixAttention different from ordinary KV caching?: Ordinary KV caching keeps K and V for an active sequence so later decode tokens do not recompute earlier tokens. RadixAttention adds cross-request reuse: it indexes cached prefixes by token sequence, finds the longest prefix shared with a new request, and attaches that request to the existing KV entries before computing only the unmatched suffix.
How is RadixAttention different from vLLM prefix caching?: RadixAttention uses a compressed tree over token sequences, so it can match and split at token-level prefix boundaries. vLLM’s prefix caching uses hashed KV blocks, so reuse happens through consecutive matching blocks. The tree gives direct longest-prefix semantics; the hash approach aligns naturally with fixed pages, immutable block keys, and high-throughput block management.
When does RadixAttention not help?: It helps least when requests do not share early tokens, when shared text is shorter than the useful cache granularity, or when KV memory is already scarce. A unique user preamble, timestamp, nonce, or retrieved passage near the start can make the longest match tiny. Then the system pays tree and allocator overhead without avoiding much prefill.
Does RadixAttention reduce the memory used by the KV cache?: Not inherently. It can reduce duplicate computation by reusing existing KV tensors, but those tensors still occupy memory while cached. In some workloads it may increase pressure because long prefixes are kept in hope of reuse. The benefit depends on whether saved prefill work outweighs metadata, fragmentation, eviction churn, and occupied KV capacity.

Every term