04.12 · Walkthrough
Benchmarking Your Server
Run a load test that is not self-deception: a realistic prompt-length distribution, controlled concurrency, percentiles rather than means, and a warm cache reported separately from a cold one.
A useful LLM serving benchmark fixes the offered load, samples realistic prompt and output lengths, and reports tail latency with token-level metrics. Cold and warm cache results must be separate, because prefix reuse, allocator state, CUDA graphs and scheduler behaviour can change both latency and throughput.
What this lesson answers
- how to benchmark an LLM server properly
- why percentiles matter in load testing
- cold cache versus warm cache benchmark
Notes
A non-self-deceptive LLM server benchmark is a controlled experiment over an arrival process, prompt/output length distribution, and concurrency cap, reporting latency percentiles and token throughput with cold-cache and warm-cache runs separated. The governing accounting is Little’s law, : if a run sustains request rate requests/s and end-to-end latency seconds, the average in-flight concurrency is . For generated tokens, report prefill time, time-to-first-token, inter-token latency, and decode throughput, not just request/s.
References
Common questions
- What makes an LLM server benchmark misleading?
- A benchmark becomes misleading when it uses a tidy synthetic workload, uncontrolled client concurrency, or average latency as the main result. Real serving mixes short turns, retrieval prompts and long contexts. The client must cap in-flight requests, and the report should show latency percentiles, token throughput, failures and cache state.
- Why should prompt length distribution be realistic?
- Prompt length strongly affects prefill cost, memory pressure and batching behaviour. If every request has the same small shape, the server may look stable while hiding the cost of long-context traffic. A better workload samples from expected production traffic, with output lengths drawn separately from input lengths.
- Why report cold and warm cache results separately?
- Cold runs include first-use effects such as empty prefix caches, allocator growth, graph capture and tokenizer misses. Warm runs measure behaviour after those paths have been primed, and may include reusable system prompts or retrieval templates. Mixing them into one number hides what users experience at startup versus steady state.
Short definition: what is Benchmarking Your Server?