01.05 · Concept
Sampling
Implement temperature scaling, top-k, top-p, and greedy sampling. Explain what each hyperparameter controls and when to use it.
Sampling turns next-token logits into an actual token by scaling confidence, pruning unlikely candidates, normalising what remains, and either drawing randomly or taking the maximum. Temperature changes entropy, top-k limits candidate count, top-p limits cumulative probability mass, and greedy decoding gives deterministic output.
What this lesson answers
- how does temperature affect token sampling
- top-k versus top-p sampling difference
- when should I use greedy decoding
Notes
Sampling converts next-token logits into one token by optionally transforming logits, filtering candidates, normalizing with softmax, then drawing or selecting. Algorithm: temperature ; top-k keeps ; top-p keeps smallest sorted set such that ; set logits outside to ; compute ; sample , or greedy uses .
Example with vocabulary and logits…
Common questions
- What does temperature do in language model sampling?
- Temperature rescales logits before probabilities are computed. Lower temperature makes the distribution sharper, so the highest scoring token dominates. Higher temperature flattens the distribution, giving lower ranked tokens more chance. A temperature near zero behaves like greedy decoding, while the default value leaves the logits unchanged.
- How are top-k and top-p sampling different?
- Top-k keeps a fixed number of highest scoring tokens, regardless of how confident the model is. Top-p keeps the smallest set of tokens whose combined probability reaches a chosen mass. That makes top-p adapt to the distribution: narrow when the model is confident, wider when many continuations are plausible.
- When is greedy decoding the right choice?
- Greedy decoding is useful when you want repeatable, low-variance output and the task has a constrained answer shape, such as extraction, classification-like generation, or regression tests. It is cheap and deterministic, but for open-ended text it can produce bland continuations or get stuck in repetitive patterns.
Short definition: what is Sampling?