Why are input and output tokens priced differently?

Input tokens are handled during prefill, where the model can process a prompt with large matrix operations over many tokens. Output tokens are generated autoregressively, so each new token requires another decode step and repeated access to model weights and KV state. The same GPU-hour therefore produces very different amounts of input and output work.

Is cost per million tokens just GPU price divided by benchmark throughput?

No. That gives a best-case floor, not a usable production price. You also need realistic utilisation after headroom, fragmentation, routing imbalance, maintenance and failed work. Then you add non-GPU costs such as CPUs, networking, logging and orchestration, plus margin and risk buffers if the number is becoming a customer-facing price.

What is commonly misunderstood about this metric?

A common mistake is averaging all tokens together. Provider-style pricing needs separate input and output costs, and often separate workload buckets as well. Cached prompts, long-context chats, short completions and high-throughput batch jobs can have different effective costs even on the same model, because they change throughput and utilisation differently.

When does the simple formula stop being reliable?

It depends on workload shape. Very small requests can be dominated by fixed scheduling and network overhead. Very long contexts can make KV-cache traffic and memory limits dominate decode. Prefix caching can make repeated input cheap, but only with strong cache hit rates. Speculative decoding helps only when draft tokens are accepted often enough.

Cost per Million Tokens

Cost per million tokens is the unit-economics measure that converts the cost of running an inference fleet into separate prices for input tokens and output tokens. It starts from GPU-hour cost, measured serving throughput and achievable utilisation, then adds overhead and margin to match the way model providers quote API pricing.

The need for this metric comes from a mismatch between how infrastructure is bought and how inference is sold. GPUs are paid for by time, while customers are charged by tokens. A fast benchmark is not enough, because idle capacity, routing gaps, retries, maintenance and reserved headroom all reduce the number of billable tokens produced per paid GPU-hour.

The calculation is an accounting transform. Measure how many input tokens per second a replica can prefill, how many output tokens per second it can decode, multiply by sustainable utilisation, and divide the GPU cost for that replica by the resulting token production rate. Input and output are priced separately because prompt processing and autoregressive generation stress the hardware in different ways.

The main trade-off is that the number looks precise while depending heavily on measurement choices. Utilisation is usually the dangerous assumption: a fleet that benchmarks well but sits partly empty has a much higher token cost. The model also breaks down for tiny requests, very long contexts, cache-heavy workloads or speculative decoding unless those cases are measured as separate traffic classes.

Engineers meet cost per million tokens when pricing an API, comparing serving stacks, choosing hardware, or explaining why output tokens cost more than input tokens. In practice it lives in a spreadsheet or capacity model fed by production traces, not vendor claims. Changes such as batching, prefix caching, quantisation or better routing should be reflected by remeasuring throughput or utilisation.

Go deeper

Inference EngineeringCost per Million TokensBuild the unit-economics model from GPU-hour price, measured tokens per second and realistic utilisation, and produce separate input and output prices the way every provider quotes them.Open the lesson

Common questions

Why are input and output tokens priced differently?: Input tokens are handled during prefill, where the model can process a prompt with large matrix operations over many tokens. Output tokens are generated autoregressively, so each new token requires another decode step and repeated access to model weights and KV state. The same GPU-hour therefore produces very different amounts of input and output work.
Is cost per million tokens just GPU price divided by benchmark throughput?: No. That gives a best-case floor, not a usable production price. You also need realistic utilisation after headroom, fragmentation, routing imbalance, maintenance and failed work. Then you add non-GPU costs such as CPUs, networking, logging and orchestration, plus margin and risk buffers if the number is becoming a customer-facing price.
What is commonly misunderstood about this metric?: A common mistake is averaging all tokens together. Provider-style pricing needs separate input and output costs, and often separate workload buckets as well. Cached prompts, long-context chats, short completions and high-throughput batch jobs can have different effective costs even on the same model, because they change throughput and utilisation differently.
When does the simple formula stop being reliable?: It depends on workload shape. Very small requests can be dominated by fixed scheduling and network overhead. Very long contexts can make KV-cache traffic and memory limits dominate decode. Prefix caching can make repeated input cheap, but only with strong cache hit rates. Speculative decoding helps only when draft tokens are accepted often enough.

Every term