Determinism
Determinism in model inference is the property that the same input, model, and serving conditions produce the same output. Temperature zero only makes token selection greedy after logits are computed; it does not guarantee identical logits, because GPU floating-point reductions can change with kernel choice, shape, routing, and batch composition.
The common misunderstanding is that setting temperature to zero removes all sources of variation. It removes sampling, not numerical path dependence. In production serving, the same request may run alone once and share a batch with longer prompts another time. That can alter tensor shapes, attention layouts, and reduction schedules, slightly changing logits. If the leading token scores are close, a tiny numerical difference can flip the greedy choice and the rest of the generation follows a different path.
The mechanism is floating-point nonassociativity in ordinary model operations. Matmuls, attention, MLPs, and normalisation layers all contain sums, and GPU kernels compute those sums in trees, tiles, split reductions, collectives, or atomics. Changing the batch or sequence grouping can choose a different tree. Since rounded floating-point addition is not associative, the final logits need not be bitwise identical. Greedy decoding then applies argmax to whatever logits that particular run produced.
Batch-invariant mode tries to make a request see the same numerical route regardless of other live traffic. Systems do this with fixed padding or bucketing, deterministic GEMM and attention kernels, stable tie-breaking, fixed collectives, and sometimes isolating decode lanes. The trade-off is real: serving gives up freedom to pack requests tightly, use autotuned kernels, and keep accelerators busy. It may increase memory traffic, reduce occupancy, and worsen latency under bursty workloads.
Engineers meet this when evaluation, regression tests, audit trails, caches, or debugging require repeatable text rather than merely similar answers. It also appears after backend changes: a FlashAttention upgrade, a TensorRT-LLM kernel choice, vLLM PagedAttention batching, prefix-aware scheduling, or disaggregated prefill and decode routing can all alter execution conditions. The honest answer is that determinism depends on which layer you control: sampler, kernels, shapes, collectives, placement, and batching.
Common questions
- Why can temperature zero still produce different outputs?
- Temperature zero makes the decoder choose the highest-logit token instead of sampling. It does not force the model to compute exactly the same logits each time. If batching or kernel selection changes the order of floating-point reductions, the logits can move slightly. When the top candidates are close, argmax can choose a different token.
- Is this a bug in the model or serving stack?
- Usually no. The kernels are doing valid floating-point arithmetic, but not necessarily in a batch-invariant order. High-throughput inference systems deliberately reshape, pack, tile, and route work to use hardware efficiently. Those choices can change rounding behaviour while remaining within normal numerical error. It becomes a bug only if the system promised stronger reproducibility.
- When is batch-invariant determinism worth the cost?
- It is worth paying for when exact repeatability affects correctness, accountability, or developer workflow: evaluation harnesses, regression testing, legal or medical audit trails, cache-keyed generation, and investigations of rare divergences. It is often not worth it for interactive products where semantic consistency is enough and throughput or tail latency matters more.
- What should I check before blaming randomness?
- Check whether the request saw the same batch shape, padding, attention kernel, GEMM algorithm, tensor-parallel collective, KV layout, replica, and routing path. Also check tie-breaking in argmax. A seed only controls random sampling; it does not freeze floating-point reduction order or the serving scheduler.