Module 08
Thinking Costs Tokens
A model that thinks before it answers moves the compute from training to serving, makes output length unpredictable, and drags reproducibility and the reinforcement-learning loop in behind it. Every number in modules 2 and 4 has to be recomputed for a request that emits thirty thousand tokens nobody reads.
12 lessons · 12 videos · 2h 2m
- 08.01
Test-Time Compute
State the trade the field made: buy accuracy with inference tokens instead of parameters, and understand why that moves the dominant cost of an AI product from a one-off training run to a bill that arrives every day.
- 08.02
What a Reasoning Model Emits
Separate reasoning tokens from answer tokens at the serving boundary: both are generated, both are decoded, both are billed, and only one is shown. Explain what that does to a token-count budget written for a chat model.
- 08.03
The Decode-Heavy Workload
Redo module 2's arithmetic for a request whose output is two orders of magnitude longer than its input, and show that a reasoning request is almost pure decode, so every memory-bandwidth conclusion applies harder, and every prefill optimisation matters less.
- 08.04
Effort and Budgets
Drive the one knob a caller is given, an effort level or a thinking-token ceiling, and plot the curve it moves along: accuracy against latency and cost. Then set it per route rather than per product, because most requests do not need it at all.
- 08.05
Unpredictable Output Length
Explain why a scheduler that plans capacity from an expected output length degrades when that length varies by an order of magnitude between requests, and describe what the batch does when a handful of very long thoughts hold their slots for minutes.
- 08.06
Parallel Sampling
Spend the budget sideways instead of deep, on best-of-n and self-consistency, and price it properly: n branches share one prompt prefix but each own their divergent KV, so the cost is sublinear in memory and linear in decode.
- 08.07
Verifiers in the Serving Path
Put a second model in the request path, a verifier or reward model that scores candidate answers, and reason about the two designs: scoring the final answer, or scoring each step. Account for the fact that the verifier is inference too.
- 08.08
Serving Two Models
Co-locate a small model beside a large one, whether a verifier, a draft model or a router, and divide the device between them: weights, KV budget and scheduling priority. Show what happens to the big model's throughput when the small one is not given its own priority class.
- 08.09
SLOs for Thinking
Rewrite module 4's latency vocabulary for a model that is silent for the first thirty seconds: time-to-first-token measures a token nobody sees, so define what to promise instead, and decide whether to stream the reasoning at all.
- 08.10
Determinism
Explain why the same prompt at temperature zero can return different text on two runs: the reduction kernels are batch-size dependent, so a request's own numerics change with who else happened to be in the batch. Know what batch-invariant mode costs and when it is worth it.
- 08.11
Rollouts Are Inference
Recognise the reinforcement-learning loop as an inference problem: the trainer spends most of its wall clock inside a serving engine generating rollouts. Work through weight synchronisation, pausing without dropping in-flight requests, and why bitwise consistency between the trainer and the sampler is a correctness requirement rather than a nicety.
- 08.12
Cost per Solved Task
Replace module 6's cost per million tokens with the only number a reasoning product can be judged on, cost per task actually solved, and use it to settle the real question: a small model sampled many times, or one call to a large one.