What a Reasoning Model Emits

Notes

A reasoning model’s “thinking” is not a side channel in the serving stack; it is ordinary autoregressive decoding whose intermediate tokens are suppressed before the HTTP response is rendered. If the model samples a hidden chain $r_{1:R}$ and then visible answer tokens $a_{1:A}$ after prompt tokens $x_{1:P}$, the served generation cost is governed by $T_{\text{gen}}=R+A$, not $A$, and the decode loop still evaluates $p(y_t\mid x_{1:P},y_{1:t-1})$ for every $t\in 1,\dots ,R+A$. The user may only see $a_{1:A}$, but the scheduler, KV cache, logits kernel, sampler, and billing meter see both $R$ and $A$. This is the serving-boundary distinction: “reasoning tokens” are hidden by policy or API schema, not skipped by the transformer. A chat-model token budget like “allow 1,000 output tokens” becomes ambiguous unless it says whether it means visible output only or total generated output. For reasoning models, the safe budget equation is $P+R+A\le C$ for context length and $R+A\le M$ for max generated tokens.

The arithmetic is brutal because hidden tokens are decode tokens, and decode for large models is often bandwidth dominated by weight reads plus KV traffic. Take a 70B FP16 model on one 80GB H100 with 3.35TB/s HBM bandwidth. The model weights alone are about $70\text{B}\times 2=140$GB, so a lower-bound single-token decode time from weight streaming is $140/3350=0.0418$s, or about $23.9$ tokens/s before attention KV, kernels, tensor-parallel communication, and scheduler overhead. Now add KV cache size for a Llama-like 70B with 80 layers, 8 KV heads under GQA, head dim 128, FP16 KV: per token it stores $2\times 80\times 8\times 128\times 2=327,680$ bytes, about $0.313$MiB. A request that shows 300 answer tokens but consumes 2,000 hidden reasoning tokens generated $2,300$ decode tokens, costing at least $2,300\times 0.0418=96.1$s of single-stream weight bandwidth and $2,300\times 0.313=720$MiB of KV for that sequence. A chat budget written as “300 output tokens” underestimates decode work by $2,300/300=7.67\times$.

This is why the serving API must account separately for visible answer tokens, hidden reasoning tokens, and prompt/prefix tokens, even when it only returns the answer text. OpenAI-style “reasoning token” usage fields, Anthropic’s extended thinking blocks, and similar APIs are exposing a quantity the inference engine already had to schedule: generated-but-not-shown tokens. In vLLM, the relevant machinery is the continuous batching and PagedAttention allocator from the vLLM paper “Efficient Memory Management for Large Language Model Serving with PagedAttention” by Kwon et al.; the hidden tokens occupy the same paged KV blocks as visible tokens. SGLang’s runtime and RadixAttention prefix cache from Zheng et al.’s SGLang work can reuse shared prompt prefixes, but it cannot reuse a private hidden trace that differs per request. TensorRT-LLM’s inflight batching, NVIDIA Dynamo’s disaggregated serving release, and llm-d’s Kubernetes-oriented serving stack all still schedule hidden reasoning as decode work because the boundary is above the model executor, not inside the matmul.

The token-count budget changes in two places: admission control and user-facing truncation. For an ordinary chat model, an engineer might reserve $P+A_{\max }\le C$ and size throughput from an expected visible completion length $E[A]$. For a reasoning model, the reservation has to be $P+R_{\max }+A_{\max }\le C$, while capacity planning uses $E[R+A]$, and the billable meter should count $R+A$ completion tokens even if the serializer hides $R$. If a product says “summarize in 200 tokens” and sets only $A_{\max }=200$, the model may still spend 5,000 hidden tokens before producing those 200 visible tokens unless a separate reasoning cap or effort setting is enforced. Conversely, if the API has a single `max_tokens=200`, the answer may be empty or abruptly truncated because the hidden trace consumed the entire generation allowance. The correct serving contract needs two knobs or one clearly documented total knob: a hidden reasoning budget $R_{\max }$ and a visible response budget $A_{\max }$, or a total completion cap $M$ with visibility caveats.

The mechanism stops helping when the marginal accuracy gain from more hidden tokens is smaller than the marginal loss from latency, cost, or context eviction. Concretely, on the 70B/H100 example, raising hidden reasoning from 500 to 4,000 tokens adds $3,500\times 0.0418=146$s of lower-bound single-stream decode time and about $3,500\times 0.313=1.10$GiB of KV for one request; under batching, it also keeps that sequence resident across many scheduler iterations and can block shorter jobs. It can make quality worse when the hidden trace crowds out retrieved evidence or conversation state in a fixed context window: with $C=8,192$, $P=6,500$, and $A_{\max }=512$, the maximum hidden budget is only $1,180$ tokens, so a 3,000-token private chain forces truncation or rejection. It also loses under speculative decoding schemes such as EAGLE, from Li et al.’s “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” when the draft model’s acceptance rate collapses on long, high-entropy reasoning traces; rejected draft tokens are extra work, not free thought.

Kernel optimizations do not change the accounting, only the constant factors. FlashAttention, from Dao et al.’s IO-aware attention papers, reduces attention memory traffic during prefill and long-context attention, but every hidden token still appends KV and advances the recurrence one position. PagedAttention reduces KV fragmentation and enables higher utilization under continuous batching, but a hidden-token-heavy request still consumes pages proportional to $R+A$. TensorRT-LLM’s paged KV cache, inflight batching, and speculative decoding plugins, SGLang’s structured generation runtime, vLLM’s OpenAI-compatible server, NVIDIA Dynamo’s disaggregated prefill/decode orchestration, and llm-d’s production deployment patterns all have to surface or internally estimate hidden completion length for scheduling. The operational lesson is narrow and specific: “answer tokens” are a rendering property, while “reasoning tokens” are generated tokens with the same decode and KV consequences; a budget inherited from chat serving is wrong unless it is rewritten around total generated tokens and then mapped back to what the user is allowed to see.