How is goodput different from throughput?

Throughput counts work completed, such as tokens per second or requests per second, whether or not the responses were timely. Goodput counts only requests that met the defined latency objective. A system can have higher throughput and lower goodput if batching or queueing makes many requests miss TTFT or TPOT targets.

Why use both TTFT and TPOT in the SLO?

They capture different user-visible failures. TTFT tells you how long the user waits before anything appears, so it is sensitive to queueing and prefill. TPOT tells you whether generation streams smoothly after it starts, so it is sensitive to decode scheduling, KV traffic, and batch shape. Meeting only one can still feel broken.

What commonly gets misunderstood about goodput?

A common mistake is treating a saturated benchmark as capacity. Saturation often measures how much work the GPUs can absorb, not how much timely service the endpoint can provide. Goodput must be measured at offered load, against explicit percentile SLOs, using the same request mix and cache conditions expected in production.

Does improving batching always improve goodput?

No. Better batching can raise goodput when it reduces idle GPU time without breaking latency budgets. It can lower goodput when it lets long prefills block short requests, increases decode step latency, or creates KV memory pressure. Whether it helps depends on the workload distribution, model bottleneck, scheduler policy, and SLO thresholds.

Goodput and SLOs

Goodput is the rate of completed inference requests that satisfy the service-level objective, usually both time to first token and time per output token, rather than the raw token rate a server emits. It turns latency promises into a capacity measure: how much offered load the system can accept while still behaving well for users.

Raw throughput is misleading for LLM serving because the GPU can look busier while users see worse latency. Larger batches, longer queues, or aggressive scheduling may raise emitted tokens per second, but some requests wait too long for their first token or receive later tokens too slowly. Goodput exists to count only the requests that stayed inside the user-visible latency budget, not all work the hardware managed to finish.

Mechanically, each completed request is tested against two gates. TTFT measures wall-clock time from admission to the first generated token, covering queueing, prefill, and the first decode step. TPOT measures the average cadence after the first token, commonly using the time from first to last token divided across the remaining generated tokens. A request contributes to goodput only if both values are within their SLO thresholds.

The trade-off is that goodput can reject optimisations that improve a lower-level metric. Bigger continuous batches may improve GPU utilisation but increase waiting time. Chunked prefill may protect short requests but add overhead if chunks are too small. Prefix caching, speculative decoding, or prefill/decode disaggregation help only when their assumptions hold. The honest answer is often workload-dependent: prompt lengths, output lengths, cache reuse, and sampling settings change the result.

Engineers meet goodput when tuning inference servers such as vLLM, SGLang, TensorRT-LLM, Dynamo, or Kubernetes-based serving stacks. The practical test is a load sweep using the real prompt and output distribution, with explicit TTFT and TPOT targets. Report the highest requests per second that meet the joint SLO, and keep warm-cache and cold-prefix cases separate so the number means something operationally.

Go deeper

Inference EngineeringGoodput and SLOsUse goodput instead of raw throughput: requests per second that met both the TTFT and the TPOT target. Set a latency budget a serving change can be measured against.Open the lesson

Common questions

How is goodput different from throughput?: Throughput counts work completed, such as tokens per second or requests per second, whether or not the responses were timely. Goodput counts only requests that met the defined latency objective. A system can have higher throughput and lower goodput if batching or queueing makes many requests miss TTFT or TPOT targets.
Why use both TTFT and TPOT in the SLO?: They capture different user-visible failures. TTFT tells you how long the user waits before anything appears, so it is sensitive to queueing and prefill. TPOT tells you whether generation streams smoothly after it starts, so it is sensitive to decode scheduling, KV traffic, and batch shape. Meeting only one can still feel broken.
What commonly gets misunderstood about goodput?: A common mistake is treating a saturated benchmark as capacity. Saturation often measures how much work the GPUs can absorb, not how much timely service the endpoint can provide. Goodput must be measured at offered load, against explicit percentile SLOs, using the same request mix and cache conditions expected in production.
Does improving batching always improve goodput?: No. Better batching can raise goodput when it reduces idle GPU time without breaking latency budgets. It can lower goodput when it lets long prefills block short requests, increases decode step latency, or creates KV memory pressure. Whether it helps depends on the workload distribution, model bottleneck, scheduler policy, and SLO thresholds.

Every term