Why is ordinary time-to-first-token misleading for reasoning models?

Because the first token may be generated for an internal reasoning buffer and then discarded or hidden before it reaches the client. The serving stack sees progress, but the user sees an empty response. For these models, time-to-first-token is mostly a kernel or decoder health metric, not a customer-facing latency promise.

Should reasoning tokens be streamed to make latency look better?

It depends on whether the product can safely expose them. Streaming reasoning makes the first visible output arrive earlier, but it may reveal private scratchpad text, unsafe content, or tentative claims that the final answer later corrects. If reasoning is suppressed, send explicit progress events instead of pretending hidden tokens are visible progress.

What should an SLO for a silent reasoning model include?

It should name the clock being promised: request acknowledgement, heartbeat cadence, time to first visible or meaningful output, time to final answer, and the policy for hidden-token budgets. It should also report hidden decode separately from visible decode, so a good visible streaming metric cannot hide a long blank interval.

SLOs for Thinking

SLOs for Thinking are latency objectives for models that generate hidden reasoning before producing user-visible output. They replace misleading time-to-first-token promises with measures such as time to first visible token, time to final answer, and hidden-token budget exhaustion, so the service contract matches what the user actually experiences.

The problem is that a reasoning model may be busy for a long time while the interface shows nothing. Classical time to first token can still look healthy, because the first decoded token might be an internal scratchpad token suppressed by the gateway. That makes dashboards, retries, load balancer timeouts, and user perception disagree. The SLO has to measure the first visible, meaningful output, not merely the first token the decoder produced.

Mechanically, a request has a prefill phase, then a sequence of hidden decode steps, then policy or filtering, then visible answer decoding. Each hidden token still runs the model and extends the request state, even though it is not streamed. A useful contract therefore separates time to acceptance, time to first visible token, time to final answer, and any maximum hidden-thinking budget enforced by the router or sampler.

The trade-off is that hidden reasoning buys answer quality only by spending decode time that the user cannot see. Suppressed tokens are not free: they consume scheduling capacity, cache space, and per-token model execution. Streaming the reasoning can reduce blank-screen perception, but it exposes unstable intermediate text, possible policy-sensitive content, and reasoning that may contradict the final answer. The right answer depends on product, safety, and latency requirements.

Engineers meet this in chat APIs, agent runtimes, inference gateways, and dashboards for models with long scratchpads or reasoning traces. The practical metrics are not just token throughput and ordinary streaming latency. You need separate accounting for hidden and visible decode tokens, heartbeats or progress events if silence is unacceptable, and SLOs that distinguish first visible token from final answer completion.

Go deeper

Inference EngineeringSLOs for ThinkingRewrite module 4's latency vocabulary for a model that is silent for the first thirty seconds: time-to-first-token measures a token nobody sees, so define what to promise instead, and decide whether to stream the reasoning at all.Open the lesson

Common questions

Why is ordinary time-to-first-token misleading for reasoning models?: Because the first token may be generated for an internal reasoning buffer and then discarded or hidden before it reaches the client. The serving stack sees progress, but the user sees an empty response. For these models, time-to-first-token is mostly a kernel or decoder health metric, not a customer-facing latency promise.
Should reasoning tokens be streamed to make latency look better?: It depends on whether the product can safely expose them. Streaming reasoning makes the first visible output arrive earlier, but it may reveal private scratchpad text, unsafe content, or tentative claims that the final answer later corrects. If reasoning is suppressed, send explicit progress events instead of pretending hidden tokens are visible progress.
What should an SLO for a silent reasoning model include?: It should name the clock being promised: request acknowledgement, heartbeat cadence, time to first visible or meaningful output, time to final answer, and the policy for hidden-token budgets. It should also report hidden decode separately from visible decode, so a good visible streaming metric cannot hide a long blank interval.

Every term