Why is output length harder to schedule than input length?

Input length is known when the request arrives, so prefill cost and initial KV use can be estimated directly. Output length is only known after generation stops. With reasoning models, many tokens may be hidden from the user but still decoded and cached. The scheduler is therefore betting on a future runtime property, not measuring a present one.

Does continuous batching solve unpredictable output length?

It helps, but it does not eliminate the tail. Continuous batching lets the server insert new requests when short ones finish, so capacity is not locked to an original static batch. However, very long requests keep their slots and KV histories for many decode iterations. The batch becomes a moving mixture of fresh short jobs around persistent long jobs.

Can PagedAttention, FlashAttention, or speculative decoding make this problem go away?

No. These techniques improve different parts of the serving path: KV allocation, attention IO, or the number of accepted tokens per model pass. They can reduce waste or speed some generations, but a long continuation is still many autoregressive steps and a long-lived KV cache. If the scheduler guessed the length badly, the admission error remains.

What practical controls reduce the damage?

Common mitigations include stricter max-token limits, reserving KV against a pessimistic length, separating short and long jobs into different queues, using backpressure earlier, and treating reasoning modes as a different scheduling class. The right choice depends on product latency goals, memory headroom, whether long generations are valuable, and how often tail requests arrive together.

Unpredictable Output Length

Unpredictable output length is a serving failure mode where an inference scheduler admits work using an expected generation length, but actual requests produce much shorter or much longer continuations. Because autoregressive decoding advances token by token, long hidden reasoning or verbose answers hold batch slots, KV memory, and GPU decode cycles far longer than planned.

The problem appears when output length has a fat tail rather than clustering near the average. A scheduler may treat a request as if it will produce a typical answer, but a reasoning model may generate a long internal chain before returning anything useful. Capacity planning then underestimates both time and memory. The result is not just a slow request. A small number of long generations can distort the whole live batch.

Autoregressive serving runs prefill for the prompt, then repeatedly decodes the next token for every live sequence until each one stops or hits a limit. During that loop, each sequence keeps its KV cache so future attention can see prior tokens. Continuous batching can add new requests as old ones finish, but long requests remain resident. Short jobs churn through the gaps while old long-context sequences become the stable core of the batch.

The trade-off is that mean-based admission gives good utilisation when requests are similar, but becomes optimistic when lengths vary by an order of magnitude. Paging KV blocks reduces allocator waste, and faster attention kernels reduce memory traffic, but neither changes the number of decode steps a long continuation requires. Conservative reservations, length caps, or length-class queues may waste capacity on ordinary traffic, yet avoid emergency backpressure, preemption, or recomputation under tail-heavy load.

Engineers meet this in LLM serving systems using continuous or in-flight batching, especially with reasoning models, tool-using agents, long-context chat, and APIs that expose generous max-token settings. It shows up as rising inter-token latency, stalled admission, KV memory pressure, and batches whose headline size looks healthy while a few old requests dominate residency. A common misunderstanding is to blame only queue length; the deeper issue is slot lifetime inside the decode loop.

Go deeper

Inference EngineeringUnpredictable Output LengthExplain why a scheduler that plans capacity from an expected output length degrades when that length varies by an order of magnitude between requests, and describe what the batch does when a handful of very long thoughts hold their slots for minutes.Open the lesson

Common questions

Why is output length harder to schedule than input length?: Input length is known when the request arrives, so prefill cost and initial KV use can be estimated directly. Output length is only known after generation stops. With reasoning models, many tokens may be hidden from the user but still decoded and cached. The scheduler is therefore betting on a future runtime property, not measuring a present one.
Does continuous batching solve unpredictable output length?: It helps, but it does not eliminate the tail. Continuous batching lets the server insert new requests when short ones finish, so capacity is not locked to an original static batch. However, very long requests keep their slots and KV histories for many decode iterations. The batch becomes a moving mixture of fresh short jobs around persistent long jobs.
Can PagedAttention, FlashAttention, or speculative decoding make this problem go away?: No. These techniques improve different parts of the serving path: KV allocation, attention IO, or the number of accepted tokens per model pass. They can reduce waste or speed some generations, but a long continuation is still many autoregressive steps and a long-lived KV cache. If the scheduler guessed the length badly, the admission error remains.
What practical controls reduce the damage?: Common mitigations include stricter max-token limits, reserving KV against a pessimistic length, separating short and long jobs into different queues, using backpressure earlier, and treating reasoning modes as a different scheduling class. The right choice depends on product latency goals, memory headroom, whether long generations are valuable, and how often tail requests arrive together.

Every term