Why is bitwise consistency treated as correctness, not polish?

Because many reinforcement-learning objectives compare the current policy with the policy that actually produced the sampled tokens. If the stored old log probability came from one implementation and the trainer recomputes against another, the ratio or KL term no longer means what the loss assumes. Small numerical differences can matter when they are systematic or correlated with selected tokens.

Can the rollout server update weights while requests are running?

Not safely for a single trajectory. A generated sequence should come from one policy snapshot, with one tokenizer, one template, one decoding configuration, and one numerical path. A common protocol is to stage the new weights, stop sending new requests to old workers, allow existing decodes to finish, then atomically route later requests to the new version.

Are normal inference speedups always valid for rollouts?

No. They are valid only if they preserve the target policy’s sampled distribution and recorded probabilities. Paged KV caching or batching may be transparent, while speculative decoding is acceptable only when its accept-reject procedure leaves the final token distribution unchanged and the logged probabilities come from the target model, not the draft model.

Rollouts Are Inference

Rollouts are inference means that reinforcement-learning post-training spends much of its time using the current policy as a serving model to generate sampled completions. Those completions, their log probabilities, and their policy version become training data, so the rollout engine must behave like a correct, versioned inference system.

The need for this idea comes from a mismatch in how teams picture reinforcement learning. The visible algorithm is a trainer computing policy-gradient updates, but the expensive wall-clock path is often prompt batches being decoded token by token. If that generation is slow, inconsistent, or poorly batched, the optimiser waits. The reinforcement-learning loop is therefore not just a training loop with some sampling attached; it is a serving workload feeding a trainer.

Concretely, a rollout server receives prompts, runs the policy snapshot autoregressively, samples tokens under configured decoding rules, and records the old log probabilities used by the later objective. The trainer consumes those fixed-version trajectories, computes rewards and advantages, then publishes a newer policy. Safe systems tag every request with a policy version, stage new weights, stop routing new work to old workers, let in-flight sequences finish, and only then retire the old version.

The trade-off is that serving optimisations become correctness risks, not just performance choices. Continuous batching, paged KV caches, fused sampling, speculative decoding, tensor-parallel layouts, and low-precision kernels can change timing, memory use, or numerical results. Some differences are harmless, but not all. If the trainer recomputes probabilities under slightly different weights, tokenisation, masks, position handling, precision, or sampling rules, policy ratios and KL terms are biased. It depends on the objective and on whether the sampled distribution is exactly preserved.

Engineers meet this in RLHF, RLAIF, PPO-like training, rejection sampling, and other post-training pipelines built around generated candidates. The relevant components look like production inference infrastructure: vLLM-style paged attention, SGLang-style runtimes, TensorRT-LLM engines, FlashAttention kernels, speculative decoding, queueing, routing, and weight-loading protocols. Debugging usually means checking version tags, old logprobs, tokenizer and chat-template identity, EOS behaviour, precision modes, and whether any request changed policy mid-sequence.

Go deeper

Inference EngineeringRollouts Are InferenceRecognise the reinforcement-learning loop as an inference problem: the trainer spends most of its wall clock inside a serving engine generating rollouts. Work through weight synchronisation, pausing without dropping in-flight requests, and why bitwise consistency between the trainer and the sampler is a correctness requirement rather than a nicety.Open the lesson

Common questions

Why is bitwise consistency treated as correctness, not polish?: Because many reinforcement-learning objectives compare the current policy with the policy that actually produced the sampled tokens. If the stored old log probability came from one implementation and the trainer recomputes against another, the ratio or KL term no longer means what the loss assumes. Small numerical differences can matter when they are systematic or correlated with selected tokens.
Can the rollout server update weights while requests are running?: Not safely for a single trajectory. A generated sequence should come from one policy snapshot, with one tokenizer, one template, one decoding configuration, and one numerical path. A common protocol is to stage the new weights, stop sending new requests to old workers, allow existing decodes to finish, then atomically route later requests to the new version.
Are normal inference speedups always valid for rollouts?: No. They are valid only if they preserve the target policy’s sampled distribution and recorded probabilities. Paged KV caching or batching may be transparent, while speculative decoding is acceptable only when its accept-reject procedure leaves the final token distribution unchanged and the logged probabilities come from the target model, not the draft model.

Every term