08.11 · Walkthrough
Rollouts Are Inference
Recognise the reinforcement-learning loop as an inference problem: the trainer spends most of its wall clock inside a serving engine generating rollouts. Work through weight synchronisation, pausing without dropping in-flight requests, and why bitwise consistency between the trainer and the sampler is a correctness requirement rather than a nicety.
Rollout generation in RLHF-style training is a serving workload: prompts are decoded under a fixed policy version, logged, scored, then consumed by the trainer. Correctness depends on versioned weight swaps, draining active requests safely, and matching trainer and sampler behaviour closely enough that stored logprobs mean what the optimiser assumes.
What this lesson answers
- why are rollouts an inference workload
- how should RL trainers synchronise sampler weights
- why does rollout determinism matter for PPO
Notes
In RLHF/RLAIF-style post-training, a “rollout” is just autoregressive inference executed under a policy snapshot, followed by scoring and a policy-gradient update. For a prompt , the sampler draws from using the same decode loop as serving, stores , and the trainer later optimises an objective such as PPO’s clipped surrogate , where .
References
Common questions
- Why treat reinforcement learning rollouts as inference rather than training?
- The rollout phase runs the policy forward autoregressively to produce tokens, using the same decode path as a serving system. The trainer consumes those trajectories later. In many post-training loops, wall clock is dominated by generating completions, not by the gradient update itself, so serving-engine behaviour directly controls training throughput.
- Can sampler weights be updated while requests are running?
- Not mid-sequence. A trajectory must come from one identifiable policy version so its sampled tokens and stored logprobs match the distribution used by the optimiser. A safe system stages the new weights, stops sending fresh work to old workers, lets active decodes finish or cancel cleanly, then flips traffic to the new version.
- Why is bitwise consistency important between sampler and trainer?
- Policy-gradient objectives compare probabilities from the current policy with probabilities recorded at sampling time. If the trainer recomputes logprobs with different kernels, masks, tokenisation, sampling rules, or precision behaviour, the probability ratio is biased. Matching decoded text is not enough; the recorded distribution must match the policy that produced the rollout.
Short definition: what is Rollouts Are Inference?