Forward Pass vs Generation
Forward pass vs generation is the distinction between computing model outputs for many already-known token positions in parallel and producing text by repeatedly predicting, choosing, and appending one next token. The first is the core training computation; the second is the sequential inference loop used by language model serving.
The distinction matters because training and serving stress hardware in different ways. During training, the target sequence is known, so the model can compute predictions for every position in a batch at once and compare them with the next tokens. During generation, the next input token does not exist until the model has produced it. That dependency turns what looks like the same neural network call into a latency-sensitive loop.
In a training forward pass, a rectangular batch of token ids flows through the transformer and produces logits for all positions. The loss is computed with teacher forcing, meaning the correct previous tokens are supplied regardless of what the model would have sampled. In generation, the prompt is first processed, then each decode step runs the model for the current last token, reads the last-position logits, applies sampling or search, appends the chosen token, and repeats.
A KV cache changes the shape of the work but not its sequential nature. Instead of recomputing attention over the whole prefix on every step, the server stores past keys and values and supplies them to later decode calls. This saves compute, but it consumes memory that grows with active sequence length, batch size, layers, and attention layout. Commonly misunderstood: inference is not simply training with backpropagation removed.
The trade-off is less parallelism over time and more complicated scheduling. Training likes large, static tensors; generation has requests with different prompt lengths, stopping points, sampling settings, and cache sizes. Throughput often depends on batching many users together, but latency suffers if the batch waits too long. The honest answer to which is cheaper depends on prompt length, generated length, cache policy, batching, and hardware.
Engineers meet this split in serving systems as prefill versus decode. Prefill processes the prompt and builds cache; decode performs many small next-token iterations. It appears in model runner APIs, schedulers, paged KV cache managers, continuous batching code, prefix caching, speculative decoding, and sampling kernels. Performance work therefore targets the generation loop, not only the raw speed of a single transformer forward computation.
Common questions
- Is generation just repeated forward passes?
- Yes, but that phrasing hides the important part. Each decode call depends on the token selected by the previous call, so the time dimension cannot be parallelised like training tokens. Serving systems also carry mutable state, especially KV cache, and must schedule many partially completed sequences rather than one fixed tensor.
- What is prefill in this distinction?
- Prefill is the initial inference pass over the prompt. It computes prompt representations and usually stores the keys and values needed for later attention. After prefill, decode steps usually consume only the newest token plus the cached context, producing one next-token distribution per active sequence.
- Why does KV cache matter for generation?
- Without KV cache, every new token would require recomputing attention information for the whole prefix again. The cache keeps past attention keys and values so decode can reuse them. That reduces repeated compute, but shifts pressure to memory capacity, memory bandwidth, cache allocation, and eviction or paging strategy.
- Why can training use more parallelism than generation?
- Training normally knows the whole input and the shifted target sequence, so predictions for many positions can be computed together. Generation does not know the future tokens. The model must choose a token, append it, then use that actual choice as part of the next input.