08.10 · Concept
Determinism
Explain why the same prompt at temperature zero can return different text on two runs: the reduction kernels are batch-size dependent, so a request's own numerics change with who else happened to be in the batch. Know what batch-invariant mode costs and when it is worth it.
Temperature zero fixes sampling, not the numeric path that produced the logits. GPU reductions can round differently when batching, kernel choice, padding, KV layout, or routing changes. If close logits swap order, greedy decoding chooses another token and the whole continuation can diverge.
What this lesson answers
- why temperature zero still gives different outputs
- how batching changes LLM inference determinism
- when batch invariant inference is worth it
Notes
Temperature-zero decoding is deterministic only after the logits are fixed; it does not guarantee that the logits are bitwise identical across serving runs. At each step the model computes projections, attention reductions, MLP reductions, layernorm/RMSNorm reductions, then selects rather than sampling from ; as , this becomes . The catch is that GPU reductions implement sums such as in a tree whose order depends on kernel choice, tile shape, sequence length grouping, and batch composition.
References
Common questions
- Why can greedy decoding return different text for the same prompt?
- Greedy decoding always picks the highest logit, but the logits are computed through floating-point reductions. Those reductions are not associative, so different kernel schedules or batch shapes can change rounding. If two candidate tokens are close, a tiny numeric change can swap the winner and send the generation down a different path.
- Does setting temperature to zero make LLM inference deterministic?
- It makes the sampler deterministic after logits are available. It does not force the model computation to produce bitwise identical logits across runs. Continuous batching, attention tiling, GEMM algorithm selection, tensor-parallel collectives, and KV cache layout can all alter arithmetic order without any random sampling being involved.
- What is the cost of batch-invariant serving?
- Batch-invariant serving restricts the runtime so a request sees the same shapes, kernels, padding, collectives, and execution lane regardless of neighbouring traffic. That improves reproducibility, but it gives up much of continuous batching’s throughput benefit and can increase memory traffic, reduce occupancy, and worsen tail latency under bursty workloads.
Short definition: what is Determinism?