08.12 · Walkthrough
Cost per Solved Task
Replace module 6's cost per million tokens with the only number a reasoning product can be judged on, cost per task actually solved, and use it to settle the real question: a small model sampled many times, or one call to a large one.
Cost per solved task is spend divided by the probability of getting a correct result. For reasoning systems, it is a better decision metric than token price because extra thinking, sampling, verification and routing are only useful when they reduce the cost of successful outcomes under real serving constraints.
What this lesson answers
- how to calculate cost per solved task
- small model many samples or large model once
- why token cost misleads reasoning model decisions
Notes
Cost per solved task is the inference metric that divides spend by success probability, not by emitted text volume. For a single attempt with input tokens , output/reasoning tokens , provider prices in dollars per token, and task accuracy , the governing quantity is . For independent samples from the same model, each costing and each solving with probability , pass@1-style solved probability becomes , so .
Common questions
- What is cost per solved task?
- Cost per solved task divides the cost of an inference policy by its chance of producing a correct answer. The policy might be one model call, repeated samples, verifier reranking, speculative decoding or routing. The metric asks what you pay for a successful user outcome, not what you pay for generated text.
- When can many small model samples beat one large model call?
- Many small samples can win when their failures are varied enough that repeated attempts materially raise the solve rate, and when serving infrastructure can batch the extra decoding efficiently. They lose when samples make the same mistakes, the verifier cannot separate good reasoning from plausible nonsense, or latency and cache pressure dominate.
- Why is cost per token a poor metric for reasoning products?
- Reasoning systems often spend more tokens deliberately to improve correctness: chain-of-thought, self-consistency, search and verification all trade extra inference for higher solve probability. Judging them by token volume alone penalises the very work that may make the product cheaper per successful task.
Short definition: what is Cost per Solved Task?