What problem does RadixAttention solve?

It avoids recomputing prefill for prompt prefixes that have already been cached. The runtime stores token sequences in a compressed radix tree and finds the longest prefix already backed by KV cache. Attention can then read the existing KV for that prefix while the model computes only the new suffix.

How is RadixAttention different from vLLM prefix caching?

RadixAttention represents token prefixes as paths in a mutable compressed tree, so it can match partial spans and answer longest-prefix queries directly. vLLM’s prefix cache works at fixed block granularity using hashed block contents, which fits naturally with paged KV management and high-throughput scheduling.

When is radix-based prefix caching a bad fit?

It is a poor fit when requests do not share early tokens, when stable template text appears after unique per-request material, or when cached prefixes are rarely reused. In those cases the system pays lookup, mutation and eviction overhead while still doing normal prefill, and the KV cache may evict more valuable state.

The KV Cache

03.09 · Concept

RadixAttention

Compare SGLang's radix tree approach to vLLM's hash-based prefix caching. Trace the radix data structure through radix_cache.py, radix_attention.py, and memory_pool.py. Two designs, different bets --- understand the tradeoffs without choosing a winner.

RadixAttention is prefix caching with a compressed token tree: shared prompt stems map to existing KV entries, so only the unmatched suffix needs prefill. Compared with vLLM’s block-hash approach, it favours fine-grained longest-prefix reuse and explicit tree operations, at the cost of mutable metadata, eviction complexity and KV memory pressure.

What this lesson answers

how does RadixAttention reuse cached prefixes
RadixAttention versus vLLM prefix caching tradeoffs
when does radix prefix caching hurt latency

Notes

RadixAttention is SGLang’s prefix-cache mechanism that stores KV-cache ownership in a compressed radix tree keyed by token sequences, so a new request can reuse the longest cached prefix instead of recomputing its prefill. For a request token sequence $x_{1 : n}$ , the reusable prefix is $m = max {k : x_{1 : k} \in T}$ , where $T$ is the set of prefixes represented by tree paths; prefill work falls from $O (n L d^{2})$ -like dense model compute for all $n$ tokens to only the suffix $O ((n - m) L d^{2})$ , while attention reads cached KV for positions $1 : m$ .

References

Common questions

What problem does RadixAttention solve?: It avoids recomputing prefill for prompt prefixes that have already been cached. The runtime stores token sequences in a compressed radix tree and finds the longest prefix already backed by KV cache. Attention can then read the existing KV for that prefix while the model computes only the new suffix.
How is RadixAttention different from vLLM prefix caching?: RadixAttention represents token prefixes as paths in a mutable compressed tree, so it can match partial spans and answer longest-prefix queries directly. vLLM’s prefix cache works at fixed block granularity using hashed block contents, which fits naturally with paged KV management and high-throughput scheduling.
When is radix-based prefix caching a bad fit?: It is a poor fit when requests do not share early tokens, when stable template text appears after unique per-request material, or when cached prefixes are rarely reused. In those cases the system pays lookup, mutation and eviction overhead while still doing normal prefill, and the KV cache may evict more valuable state.

Short definition: what is RadixAttention?

03.08 · The KV CachePrefix Caching 03.10 · The KV CacheHit Rate