The Generation Loop
01.11 · Walkthrough
Your Baseline
Run a benchmark on your own hardware: throughput, time-to-first-token, inter-token latency. Establish a reproducible measurement before any optimisation.
A serving baseline is a repeatable measurement of throughput, time to first token, and inter-token latency on your actual hardware. It fixes the workload, warms the system, timestamps streamed output with a monotonic clock, and records percentiles so later changes can be judged against the same conditions.
What this lesson answers
- how to benchmark LLM inference on my hardware
- what should an LLM serving baseline measure
- how to measure time to first token
Notes
A baseline benchmark is the reproducible measurement of an LLM server’s core generation metrics before optimization: throughput, time-to-first-token (TTFT), and inter-token latency (ITL). For each request , record timestamps , , token times , and ; compute , for , .
Common questions
- What metrics should I record before optimising an LLM server?
- Record output throughput, time to first token, and inter-token latency. Throughput tells you total token production rate, time to first token captures prompt processing and queueing, and inter-token latency shows decode responsiveness. Report percentiles for the latency metrics, not just averages, because batching and scheduling often create long tails.
- Why do I need to warm up before benchmarking inference?
- Early requests often include one-off costs such as memory allocation, kernel selection, cache setup, and graph capture. If those requests are included, the result measures start-up behaviour rather than steady serving performance. Run a warmup pass, discard it, then measure the fixed workload used for comparison.
- What must stay fixed between LLM benchmark runs?
- Keep the model, quantisation, prompt and output lengths, concurrency, sampling settings, tokenizer, batching limits, hardware, driver, runtime versions, and server flags unchanged. Otherwise you cannot tell whether a result changed because of the intended modification or because the workload or environment moved underneath it.