04.11 · Concept
Goodput and SLOs
Use goodput instead of raw throughput: requests per second that met both the TTFT and the TPOT target. Set a latency budget a serving change can be measured against.
Goodput is the request rate that still satisfies both first-token and per-output-token latency targets. Raw token throughput can improve while user-visible capacity gets worse, especially when batching, prefill, decode cadence, and KV memory pressure shift latency into the wrong part of the request lifecycle.
What this lesson answers
- how to measure LLM goodput versus throughput
- why batching can reduce user visible capacity
- how to set TTFT and TPOT SLOs
Notes
Goodput for an LLM endpoint is the arrival-rate-normalized count of requests that satisfy the service-level objective, not the number of tokens the GPUs emitted. For a trace window with completed requests , define , usually reported as requests/s at offered load ; equivalently, if requests arrive in , .
Common questions
- What is goodput for an LLM serving endpoint?
- Goodput is the rate of completed requests that meet the latency contract, not the total token rate emitted by the hardware. A request only counts if its time to first token and its per-output-token time are both within the chosen targets. This makes the metric reflect usable capacity rather than saturated GPU activity.
- Why is raw throughput misleading for batch scheduler changes?
- A larger or smarter batch can raise GPU utilisation and token output while delaying some requests past their first-token or decode cadence budget. That is a bad serving result even if tokens per second improved. The scheduler should be judged by how much offered load it can accept while keeping requests inside both latency bounds.
- How should an LLM latency budget be structured?
- Split the budget into queueing, prefill, first decode, and steady decode time. Time to first token mostly exposes queueing and prefill behaviour, while TPOT captures decode cadence after generation starts. Any serving change should state which part of the budget it spends or saves, then be tested under the expected prompt and output distribution.
Short definition: what is Goodput and SLOs?