Benchmarking Your Server
Benchmarking your server is a controlled load test that measures how an inference server behaves under a specified request mix, concurrency limit, and cache state. For LLM serving, it must report latency percentiles, token throughput, prompt and output length effects, and cold versus warm runs separately, or the result is usually misleading.
The problem is that LLM servers are easy to benchmark falsely. A single fixed prompt size, unbounded client concurrency, or an average latency number can make a fragile setup look healthy. Real traffic has short chats, retrieval-heavy prompts, occasional long contexts, varied output lengths, cache hits, cache misses, failures, and queueing. If the benchmark does not model those features, it mostly measures a toy workload rather than the server users will actually hit.
A useful benchmark fixes the workload shape before the run. The client samples prompts and output limits from a trace or an explicit mixture, keeps a chosen number of requests in flight, and replaces each completed request with another until the run ends. The report separates prefill, time to first token, per-token decode latency, end-to-end latency, output tokens per second, failures, and cancellations. Little’s law is the sanity check: concurrency, arrival rate, and latency must agree.
The trade-off is that honest benchmarks are less tidy. There is no single throughput number that describes all workloads, because prompt length, active batch size, KV cache size, scheduler policy, and prefix reuse all change the cost per token. Percentiles are noisier than means, but more useful. Warm-cache results can be legitimate for products with stable prefixes, while cold-cache results matter for first-time prompts. Mixing them together is a common mistake.
Engineers meet this when evaluating vLLM, SGLang, TensorRT-LLM, Triton, Dynamo-style disaggregated serving, Kubernetes inference stacks, or their own gateway in front of them. In practice it means writing a JSONL or trace-driven workload, pinning model revision and serving flags, sweeping fixed concurrency levels, watching client-side bottlenecks, and publishing enough configuration for someone else to reproduce the result. A benchmark without those details is usually just a graph.
Common questions
- Why are averages not enough for server benchmarking?
- Averages hide queueing and tail latency. An LLM server can have an acceptable mean while some users wait much longer because their request landed behind long prefills or large active batches. Percentiles such as p50, p95, and p99 show the distribution shape, which is what matters for user-facing latency and capacity planning.
- What is the difference between cold-cache and warm-cache benchmarking?
- A cold run starts before caches, allocator pools, compiled kernels, CUDA graphs, prefix caches, and tokenizer paths have been exercised. A warm run deliberately primes some of those effects first. Neither is universally correct. Cold is closer to first touch or bursty workloads; warm is closer to steady traffic with repeated prefixes. They should be labelled separately.
- How should concurrency be controlled in a load test?
- The client should maintain a fixed number of outstanding requests, not create unlimited coroutines and let queues grow accidentally. When a request completes, the client starts another, keeping the in-flight level constant. Sweeping that level shows where throughput stops improving and latency percentiles begin to rise sharply, which is the practical saturation point.
- What makes an LLM server benchmark realistic?
- It uses the prompt and output length distribution the service expects, includes long-tail contexts, records failures and cancellations, reports token-level timings, and states cache policy and serving configuration. The honest answer is workload-dependent: chat, RAG, code generation, and document upload traffic stress different parts of the server.