04.09 · Concept
TTFT, TPOT, ITL
Define time to first token, time per output token and inter-token latency, and say which one a user actually feels in a streaming chat versus a batch job.
TTFT is the wait until generation visibly starts, TPOT is the average decode cost per output token, and ITL is the gap between emitted tokens. Streaming chat users feel TTFT first and then ITL smoothness. Offline batch users mostly care about total completion time and output throughput.
What this lesson answers
- what is TTFT in model serving
- TTFT versus TPOT versus inter token latency
- which latency metric matters for streaming chat
Notes
Time to first token (TTFT) is the wall-clock interval from a request becoming runnable at the serving system to the moment its first generated token is emitted: , often decomposed as . Time per output token (TPOT) is a throughput-normalized decode cost, usually for a single request after the first token, or the reciprocal of system output throughput under a batch.
References
Common questions
- What is the difference between TPOT and ITL?
- TPOT is an average or throughput-oriented measure of output token cost, often useful for comparing serving efficiency. ITL is the actual sequence of gaps between tokens as they are emitted. A system can report good TPOT while a streaming user still sees uneven pauses, because batching and bursty emission hide per-user cadence.
- Which metric does a chat user actually notice?
- A chat user notices TTFT first: the delay before the response appears to start. After that, they notice ITL: whether the stream feels steady or stalls between tokens. TPOT is still useful operationally, but it is a summary and can miss the visible smoothness of the streamed response.
- Why can batching improve throughput but worsen latency?
- Batching lets the server share expensive decode work across active requests, improving aggregate tokens per second and lowering reported TPOT. The trade-off is that new requests may wait behind queued work, prefill, or decode iterations. That can increase TTFT, and larger decode steps can make each user’s token gaps longer.