SGLang: Efficient Execution of Structured Language Model Programs
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al.2024NeurIPS 2024
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Efficient Memory Management for Large Language Model Serving with PagedAttention. PagedAttention shows how to stop wasting KV memory inside one serving engine. SGLang shows the next win: stop recomputing the same prefixes across the programs your product actually runs. Few shot prompts, tool loops, RAG templates, JSON constraints, multi turn chats, and agent branches are not isolated completions. They are trees with shared roots. The mistake is treating this as a nicer prompting API. The API matters because it exposes structure the runtime can exploit, especially RadixAttention for prefix reuse and faster constrained decoding. If you are building an agent service, this paper changes what you measure. Throughput is no longer just batch size and tokens per second. It is how much repeated context you avoid doing twice.
What to take away
- RadixAttention turns shared prompt prefixes across requests into reusable KV cache instead of repeated prefill work.
- The frontend is useful because it tells the runtime where generation, branching, and parallel calls are hiding.
- Structured output is treated as a decoding systems problem, not just a prompt engineering convention.
Reads with
- Efficient Memory Management for Large Language Model Serving with PagedAttention
pagedattention fixes kv memory, sglang exploits shared kv prefixes across calls
- ReAct: Synergizing Reasoning and Acting in Language Models
agent loops are the structured programs sglang turns into reusable execution trees
- Toolformer: Language Models Can Teach Themselves to Use Tools
tool use creates repeated prompt scaffolds that radixattention avoids recomputing