06.11 · Walkthrough
Cost per Million Tokens
Build the unit-economics model from GPU-hour price, measured tokens per second and realistic utilisation, and produce separate input and output prices the way every provider quotes them.
Cost per million tokens comes from GPU-hour cost divided by measured serving throughput, corrected for real utilisation. Input and output need separate prices because prefill and decode stress the hardware differently. A usable model prices each routed traffic class from traces, then adds overhead, margin and risk buffers.
What this lesson answers
- how to calculate cost per million tokens
- why are input and output token prices different
- how does GPU utilisation affect inference pricing
Notes
Cost per million tokens is the accounting transform from a measured serving rate into the two prices a provider publishes: input /MTok. For one homogeneous GPU pool, start with fully loaded GPU-hour cost in n_gT_{in}T_{out}u$ after headroom, fragmentation, routing imbalance, retries, and maintenance.
Common questions
- Why do providers quote separate input and output token prices?
- Input tokens are handled during prefill, which is dominated by large matrix operations over the prompt. Output tokens are produced by autoregressive decode, where each new token repeatedly touches model weights and KV state. The same GPU-hour therefore produces very different amounts of input and output work, so one blended token price hides the actual cost.
- What numbers do I need to price inference from first principles?
- You need the GPU-hour cost, the number of live GPUs per serving replica, measured input-token throughput, measured output-token throughput, and sustainable utilisation after operational losses. Those losses include headroom, uneven routing, maintenance, retries and cache fragmentation. From there, compute separate floors for input and output before adding non-GPU overhead and margin.
- Why is utilisation the risky variable in token pricing?
- Benchmarks usually show what the system can do when it is kept busy under controlled conditions. A production fleet loses capacity to traffic shape, admission limits, routing imbalance, failed requests and cache behaviour. Utilisation turns theoretical tokens per second into billable tokens per second, so a small mistake there directly changes the unit economics.
Short definition: what is Cost per Million Tokens?