08.05 · Concept
Unpredictable Output Length
Explain why a scheduler that plans capacity from an expected output length degrades when that length varies by an order of magnitude between requests, and describe what the batch does when a handful of very long thoughts hold their slots for minutes.
Output length variance breaks mean-based capacity planning because decode time, KV residency, and slot lifetime follow the realised continuation, not the estimate. A few long reasoning outputs can pin batch positions for minutes, leaving short requests to rotate through reduced capacity while latency and memory pressure rise.
What this lesson answers
- why expected output length breaks inference scheduling
- how long reasoning outputs affect continuous batching
- why thinking tokens increase KV memory pressure
Notes
Unpredictable output length is the failure mode of continuous batching in which the scheduler admits requests using an estimate , but the service time is proportional to the realized decode length , including hidden “thinking” tokens for reasoning models. For an autoregressive decoder, a useful first-order occupancy equation is , where is the live batch and is the current KV-resident context.
References
Common questions
- Why is expected output length a bad scheduling input?
- Expected length works only when actual continuations stay close to the mean. With reasoning models, one request may stop quickly while another generates an order of magnitude more hidden and visible tokens. The scheduler admits work as if service time were predictable, but decode iterations and KV lifetime are paid on the realised length.
- What happens to a batch when some requests run very long?
- The long requests remain resident while shorter requests finish and are replaced around them. The batch still looks active, but its core is old sequences with growing KV histories. New work gets less effective capacity, and each decode step must continue carrying those long contexts until they finally stop or hit a limit.
- Do paged attention or speculative decoding remove this problem?
- They reduce specific costs, but they do not make an unexpectedly long continuation disappear. Paged KV management improves allocation and fragmentation; speculative decoding can reduce target-model steps when drafts are accepted. The request still occupies a slot, keeps KV live, and extends the batch for as long as generation continues.
Short definition: what is Unpredictable Output Length?