Is sampling the same as softmax?

No. Softmax converts logits into probabilities, but sampling is the larger decision procedure around that conversion. It may first apply temperature, remove candidates with top-k or top-p, then softmax only the remaining logits. The final step is either a random categorical draw or a deterministic argmax.

When should I use greedy instead of random sampling?

Use greedy decoding when you want stable, reproducible, classification-like behaviour, such as extraction, routing, or tests. It chooses the highest-scoring token at each step and avoids random variation. For creative or open-ended generation, greedy is often too conservative and can amplify repetition because it never explores plausible alternatives.

Should I tune temperature, top-k, or top-p first?

It depends on the task and the distribution shape. Temperature changes how peaked the whole distribution is. Top-k limits the number of candidates directly. Top-p limits retained probability mass and adapts to sharp or flat distributions. In practice, systems often apply temperature first, then filtering, then softmax and sampling.

If I set a random seed, will sampling be exactly reproducible?

Not always. A seed fixes the random-number stream only if the same operations consume it in the same order. In batched serving, request scheduling, sequence lengths, retries, and implementation details can change how random states advance. For strict reproducibility, you must control both the seed and the execution path.

Sampling

Sampling is the step in text generation that turns a model’s next-token logits into the actual next token. It may rescale logits with temperature, discard unlikely candidates with top-k or top-p, normalise the survivors with softmax, then either draw randomly from that distribution or choose the maximum for greedy decoding.

A language model does not directly output a word. At each generation step it outputs a score for every token in the vocabulary. Those scores are not yet a decision, and always taking the highest score can make output repetitive, overly safe, or unsuitable for open-ended tasks. Sampling exists because generation needs a controllable way to trade determinism against variety while still respecting what the model considered plausible.

Concretely, the sampler receives a vector of logits after a forward pass. Temperature divides those logits before softmax: lower temperature sharpens the distribution, while higher temperature flattens it. Top-k keeps only the highest-scoring candidates. Top-p sorts candidates by probability and keeps the smallest prefix whose cumulative probability reaches the chosen mass. Everything outside the retained set is masked, the rest is renormalised, and a categorical draw selects the token.

The trade-off is that every control changes failure modes. Greedy decoding is reproducible and simple, but can get stuck in dull or repeated continuations. Top-k is easy to reason about, but a fixed candidate count can be wrong for both sharp and flat distributions. Top-p adapts to the distribution, but needs sorting or partial sorting. Temperature can improve variety, but too much makes low-quality tokens more likely.

Engineers meet sampling inside the generation loop of inference servers, immediately after logits are produced and before the selected token is appended to each active sequence. It appears as sampling parameters in APIs and as sampler, logits processor, or decoding-kernel code in serving stacks. Reproducibility is commonly misunderstood: a seed helps, but batching, scheduling, and random-number state advancement can still affect exact outputs.

Go deeper

Inference EngineeringSamplingImplement temperature scaling, top-k, top-p, and greedy sampling. Explain what each hyperparameter controls and when to use it.Open the lesson

Common questions

Is sampling the same as softmax?: No. Softmax converts logits into probabilities, but sampling is the larger decision procedure around that conversion. It may first apply temperature, remove candidates with top-k or top-p, then softmax only the remaining logits. The final step is either a random categorical draw or a deterministic argmax.
When should I use greedy instead of random sampling?: Use greedy decoding when you want stable, reproducible, classification-like behaviour, such as extraction, routing, or tests. It chooses the highest-scoring token at each step and avoids random variation. For creative or open-ended generation, greedy is often too conservative and can amplify repetition because it never explores plausible alternatives.
Should I tune temperature, top-k, or top-p first?: It depends on the task and the distribution shape. Temperature changes how peaked the whole distribution is. Top-k limits the number of candidates directly. Top-p limits retained probability mass and adapts to sharp or flat distributions. In practice, systems often apply temperature first, then filtering, then softmax and sampling.
If I set a random seed, will sampling be exactly reproducible?: Not always. A seed fixes the random-number stream only if the same operations consume it in the same order. In batched serving, request scheduling, sequence lengths, retries, and implementation details can change how random states advance. For strict reproducibility, you must control both the seed and the execution path.

Every term