Is the generation loop the same as decoding?

They are closely related but not identical. The generation loop is the whole repeated control flow: forward pass, choose token, append, stop check, repeat. Decoding is usually the token-choice part inside that loop, such as greedy selection, temperature sampling, top-k filtering, or other rules that convert logits into the next token.

Why is generation slower than a normal forward pass?

Because generated tokens depend on earlier generated tokens. The model cannot compute all future positions independently before knowing which tokens were selected. A long prompt can be processed in a prefill pass, but the answer itself is typically produced one token at a time. KV caching reduces repeated attention work, but it does not remove the sequential dependency.

What does the KV cache change in the loop?

The KV cache stores attention keys and values computed for earlier tokens. During decode, the model can feed the newest token and attend it to cached history instead of recomputing the entire prefix. The loop still samples and appends one token at a time, but the forward pass becomes much less wasteful at the cost of extra memory.

When does the loop stop?

It depends on the serving policy and model output. Common stop conditions include an end-of-sequence token, a configured maximum number of new tokens, a matched stop string, cancellation by the client, or safety and routing logic in the server. The important point is that stopping is checked after each newly selected token.

Generation Loop

A generation loop is the autoregressive inference loop used by a language model to produce text: run a forward pass on the current tokens, turn the final-position logits into a next-token choice, append that token, and repeat until a stop token or length limit is reached.

The need for a generation loop comes from how decoder-only language models are trained and used. They do not produce a whole answer in one independent operation. Given a prefix, the model predicts a distribution for the next token. Once a token is chosen, that choice becomes part of the prefix and changes the next prediction. Text generation is therefore sequential, even when the model’s internal matrix operations are highly parallel.

Concretely, the loop keeps a tensor of token ids. Each iteration feeds the current context to the model, takes only the logits for the last position, applies decoding rules such as temperature, greedy choice, or top-k filtering, samples or selects one token, and concatenates it to the token tensor. In a simple implementation, the whole growing context is passed again each time. In production, prefill processes the prompt once, then decode advances token by token using cached attention state.

The main cost is that generation is latency-bound by this repeated dependency: the next step cannot start until the current token has been chosen. Without a KV cache, each step recomputes attention over the prefix, wasting work. With a cache, memory use rises because keys and values must be retained per request and layer. Decoding choices also trade determinism for diversity; temperature and sampling can help fluency, but can also make outputs less reproducible.

Engineers meet the generation loop in model `generate` functions, inference servers, batching schedulers, sampler code, and runtime kernels. The common misunderstanding is to think the model writes an answer in one pass. It usually does not. Serving systems preserve the same loop shape but hide it behind request queues, prefill and decode phases, stop conditions, random number state, KV-cache management, and batching of requests that finish at different times.

Go deeper

Inference EngineeringThe Generation LoopTrace the autoregressive generation loop: forward pass → sample token → append → repeat. Read Karpathy's generate function in nanoGPT.Open the lesson

Common questions

Is the generation loop the same as decoding?: They are closely related but not identical. The generation loop is the whole repeated control flow: forward pass, choose token, append, stop check, repeat. Decoding is usually the token-choice part inside that loop, such as greedy selection, temperature sampling, top-k filtering, or other rules that convert logits into the next token.
Why is generation slower than a normal forward pass?: Because generated tokens depend on earlier generated tokens. The model cannot compute all future positions independently before knowing which tokens were selected. A long prompt can be processed in a prefill pass, but the answer itself is typically produced one token at a time. KV caching reduces repeated attention work, but it does not remove the sequential dependency.
What does the KV cache change in the loop?: The KV cache stores attention keys and values computed for earlier tokens. During decode, the model can feed the newest token and attend it to cached history instead of recomputing the entire prefix. The loop still samples and appends one token at a time, but the forward pass becomes much less wasteful at the cost of extra memory.
When does the loop stop?: It depends on the serving policy and model output. Common stop conditions include an end-of-sequence token, a configured maximum number of new tokens, a matched stop string, cancellation by the client, or safety and routing logic in the server. The important point is that stopping is checked after each newly selected token.

Every term