Why is time-to-first-token misleading for reasoning models?

Because the first decoded token may be an internal reasoning token that is never shown to the user. The service can report fast TTFT while the chat window stays empty. For user experience and timeout behaviour, the useful clock starts when the first visible, meaningful answer content appears.

Should hidden reasoning be streamed to reduce perceived latency?

Only if the product is willing to expose intermediate reasoning as user-visible output. That can reduce blank-screen time, but it may reveal unstable, policy-sensitive, private, or contradictory content. If reasoning stays hidden, stream explicit progress events instead and measure them separately from model token latency.

What should an SLO include for a silent reasoning model?

It should name request acknowledgement, heartbeat or progress cadence, time to first visible token, time to final answer, and the hidden-token budget policy. Hidden and visible decode should be reported separately, otherwise dashboards can look healthy while users wait through a long silent reasoning phase.

Thinking Costs Tokens

08.09 · Concept

SLOs for Thinking

Rewrite module 4's latency vocabulary for a model that is silent for the first thirty seconds: time-to-first-token measures a token nobody sees, so define what to promise instead, and decide whether to stream the reasoning at all.

Time to first visible token is the latency promise that matters for hidden-reasoning models. Internal scratchpad tokens can make classic time-to-first-token look healthy while users see a blank UI. Reliable SLOs separate acknowledgement, hidden decode, first meaningful output, final answer, and any budget that cuts reasoning short.

What this lesson answers

what replaces TTFT for hidden reasoning models
should reasoning tokens be streamed to users
how to set SLOs for silent model thinking

Notes

A “thinking” SLO separates hidden-token generation from user-visible generation: if the model emits $r$ reasoning tokens that are suppressed and then $a$ answer tokens that are streamed, classical time-to-first-token is $T T F T = t_{1} - t_{0}$ for the first decoded token, but the user experiences $T T F V = t_{r + 1} - t_{0}$ , time to first visible token. The governing latency decomposition for a single request is $T_{visible} = T_{prefill} + \sum_{i = 1}^{r} T_{decode, i}^{hidden} + T_{policy} + T_{decode, r + 1}^{visible}$ , while completion latency is…

References

vLLM: metrics

Common questions

Why is time-to-first-token misleading for reasoning models?: Because the first decoded token may be an internal reasoning token that is never shown to the user. The service can report fast TTFT while the chat window stays empty. For user experience and timeout behaviour, the useful clock starts when the first visible, meaningful answer content appears.
Should hidden reasoning be streamed to reduce perceived latency?: Only if the product is willing to expose intermediate reasoning as user-visible output. That can reduce blank-screen time, but it may reveal unstable, policy-sensitive, private, or contradictory content. If reasoning stays hidden, stream explicit progress events instead and measure them separately from model token latency.
What should an SLO include for a silent reasoning model?: It should name request acknowledgement, heartbeat or progress cadence, time to first visible token, time to final answer, and the hidden-token budget policy. Hidden and visible decode should be reported separately, otherwise dashboards can look healthy while users wait through a long silent reasoning phase.

Short definition: what is SLOs for Thinking?

08.08 · Thinking Costs TokensServing Two Models 08.10 · Thinking Costs TokensDeterminism