What is a reasoning effort budget?

It is a caller-controlled limit on hidden generation before the visible answer is produced. The model may use that budget to plan, check or derive an answer. Those tokens still consume decode time, cache space and billable output-side usage, so the budget must be treated as production capacity, not as free internal work.

Should reasoning effort be configured globally for a product?

No. Different routes have different accuracy curves and latency tolerance. Extraction, rewriting, moderation and simple support answers often gain little from hidden reasoning. Code repair, symbolic reasoning and multi-hop synthesis may benefit, but only after measurement shows the extra solved cases justify the extra milliseconds and spend.

How do you choose the right thinking-token ceiling?

Run the same route-specific evaluation set at several effort levels, then record success rate, latency, visible output and billable usage. Pick the smallest budget where the next increase stops paying for itself. If the curve is flat, set the budget to zero and improve retrieval, tools, prompts or validators instead.

Thinking Costs Tokens

08.04 · Walkthrough

Effort and Budgets

Drive the one knob a caller is given, an effort level or a thinking-token ceiling, and plot the curve it moves along: accuracy against latency and cost. Then set it per route rather than per product, because most requests do not need it at all.

Effort budgets are hidden decode-token limits that trade accuracy for latency and cost. The useful control is not a global high-reasoning switch, but a per-route setting chosen from measured curves. Many production routes should use no hidden reasoning because validation, retrieval, tools or deterministic logic solve the actual bottleneck.

What this lesson answers

how to set reasoning effort per API route
when do thinking tokens improve model accuracy
how hidden reasoning tokens affect latency and cost

Notes

An “effort” knob is an inference-time cap on hidden reasoning tokens before the model is allowed to emit the user-visible answer: if prompt length is $p$ , hidden thinking tokens are $r$ , visible output tokens are $o$ , and price/latency are charged on generated tokens, the first-order budget equation is $T_{total} = p + r + o$ and $C = c_{in} p + c_{out} (r + o)$ . For a decoder-only transformer in steady-state decode, the latency added by effort is approximately $Δ t = r / t_{decode}$ until KV-cache or scheduler effects dominate.

References

Common questions

What is a reasoning effort budget?: It is a caller-controlled limit on hidden generation before the visible answer is produced. The model may use that budget to plan, check or derive an answer. Those tokens still consume decode time, cache space and billable output-side usage, so the budget must be treated as production capacity, not as free internal work.
Should reasoning effort be configured globally for a product?: No. Different routes have different accuracy curves and latency tolerance. Extraction, rewriting, moderation and simple support answers often gain little from hidden reasoning. Code repair, symbolic reasoning and multi-hop synthesis may benefit, but only after measurement shows the extra solved cases justify the extra milliseconds and spend.
How do you choose the right thinking-token ceiling?: Run the same route-specific evaluation set at several effort levels, then record success rate, latency, visible output and billable usage. Pick the smallest budget where the next increase stops paying for itself. If the curve is flat, set the budget to zero and improve retrieval, tools, prompts or validators instead.

Short definition: what is Effort and Budgets?

08.03 · Thinking Costs TokensThe Decode-Heavy Workload 08.05 · Thinking Costs TokensUnpredictable Output Length