Cost per Solved Task
Cost per solved task is an inference metric that divides the cost of an attempt policy by the probability that the policy actually produces a correct result. It measures reasoning systems by successful outcomes rather than token volume, so extra sampling, longer thinking, verification, or larger models are judged by whether they buy enough additional success.
Token price is a misleading headline metric for reasoning products because the whole point of reasoning-time compute is to spend more work when it improves the answer. Chain-of-thought, self-consistency, verifier reranking, speculative drafts, and retries all deliberately increase generated tokens. Calling them expensive purely because they emit more text misses the actual product question: did the extra inference turn failed user tasks into solved ones cheaply enough?
The calculation is simple: estimate the cost of a policy, then divide by its measured solve probability. For one model call, the cost includes input tokens, output or reasoning tokens, and the provider or serving price for each. For repeated samples, the success probability is not the single-call accuracy multiplied by the sample count. It is the chance that at least one sample is correct, followed by whatever voting or verifier step can select it.
The trade-off is that extra attempts have diminishing returns, especially when failures are correlated. Sampling a small model many times works only if the samples make different mistakes and the selector can identify the good trace. It fails when the model repeats the same wrong proof, lacks the needed capability, or produces plausible nonsense that fools the verifier. More tokens can then raise latency, queueing, and spend without raising solved-task probability enough.
Engineers meet this metric when choosing between one large call, many small calls, a verifier pipeline, speculative decoding, or a route that escalates hard cases. The honest answer is empirical: measure cost from invoices or GPU amortisation, measure success on tasks with unambiguous grading, then compare policies under latency and capacity constraints. Serving systems such as vLLM, SGLang, TensorRT-LLM, Dynamo, and llm-d matter because they change the cost per attempt.
Common questions
- How is cost per solved task different from cost per million tokens?
- Cost per million tokens prices text production. Cost per solved task prices successful outcomes. A policy that generates many traces may look worse by token volume but better if it solves far more tasks. Conversely, a cheap short answer is not economically good if it usually fails and forces retries, escalation, or user abandonment.
- Does sampling a smaller model many times usually beat one large model call?
- It depends on independence, selection quality, and serving efficiency. Many small samples can win when their errors are diverse, at least one good answer is often present, and voting or a verifier can find it. A large model wins when it has a capability advantage, when small-model failures are shared, or when latency and batching make many samples costly.
- What should count as a solved task?
- Use the product’s real success condition, not a proxy such as answer length or model confidence. For code, that might be passing tests. For maths, a verified final answer. For an agent, completing the requested workflow. The metric is only as trustworthy as the grading, so ambiguous or subjective tasks need careful evaluation design.
- Why do inference serving details affect this metric?
- They change the cost side of the equation. Batching, KV-cache management, prefix reuse, attention kernels, and speculative decoding can make additional samples cheaper or faster without changing what counts as success. That is why cost per solved task should be measured for the deployed policy, not inferred from a model card price alone.