The training-inference mismatch and FP16

Context. Reinforcement-learning (RL) fine-tuning of a language model runs two programs on the same weights. An inference enginesoftware built for fast generation, such as vLLM or SGLang, that samples the responses used for training; this note calls it the sampler generates the responses, and a training enginesoftware that computes losses and gradients, such as PyTorch FSDP, DeepSpeed or Megatron; this note calls it the trainer recomputes their token probabilities and takes the gradient. In exact arithmetic the two compute the same policy. In practice they use different kernels, operation orders and parallel layouts, so they round at different places and give the same token different probabilities. This training-inference mismatchthe difference between the token probabilities that the sampler and the trainer assign to the same response under the same weights makes nominally on-policy training slightly off-policy, and it has been linked to sudden collapses of RL runs. Earlier fixes reweight the gradient by importance samplingreweighting samples drawn from one distribution by the ratio of a target distribution's probability to the sampling distribution's, so that averages estimate expectations under the target ratios between the two engines, per token (Yao et al., 2025) or per response (Liu et al., 2025), both blog posts, or align the two implementations by hand. Qi et al. (2025) trace most of the mismatch to the number format: BF16bfloat16, a 16-bit floating-point format with 8 exponent bits and 7 mantissa bits: the range of 32-bit floats with coarse precision; the default for modern training, the 16-bit format that nearly all large-model training now uses.

BF16 has only 7 mantissa bitsthe bits of a floating-point number that store its significant digits; with $m$ of them, rounding to nearest changes a number by at most $2^{-(m+1)}$ of its size, so each rounding can change a value by up to $2^{-8}\approx0.4\%$, and two engines that round at different places end up with visibly different log-probabilities. The differences add up along a response: in BF16 the trainer's log-probability of a sampled response is lower than the sampler's by about one nat per thousand tokens on average (the slope of a line the paper fits across responses). FP16IEEE half precision: 5 exponent bits and 10 mantissa bits, so rounding is 8 times finer than in BF16, but the largest number is about 65,504 and the smallest normal one about $6\times10^{-5}$ rounds eight times more finely. Its narrow range matters little in fine-tuning, where pretraining has already set the scale of weights and activations, and loss scalingmultiplying the loss by a large factor $S$ before backpropagation and dividing the gradients by $S$ before the update, so that small gradients do not round to zero in FP16; dynamic versions adjust $S$ when overflows occur keeps small gradients representable. With both engines in FP16 the estimated KL divergence between their response distributions is 24 times smaller, and the plain importance-weighted policy gradient trains stably to near-perfect accuracy on a problem set the model can learn to solve, where every BF16 method in the comparison either collapses or levels off lower. The change is a configuration setting, and it also shrinks the deployment gapthe loss from tuning the weights to maximize reward under the trainer's probabilities when answers at evaluation and deployment are generated with the sampler's probabilities, which reweighting cannot remove.

Setting. A prompt $x$ gets a response $y=(y_1,\dots,y_{|y|})$ with reward $R(x,y)$, and $s_t$the prefix $(x,y_1,\dots,y_{t-1})$ before token $t$ is the prefix before token $t$. Write $\mu_\theta$the sampler's policy at weights $\theta$: $\mu_\theta(y_t\mid s_t)$ is the probability the inference engine gives token $y_t$ after $s_t$ and $\pi_\theta$the trainer's policy at weights $\theta$: the probabilities the training engine computes and differentiates for the sampler's and the trainer's policies at weights $\theta$, $\theta'$the weights that generated the current batch; $\theta$ are the weights being updated, equal to $\theta'$ before the first update on that batch for the weights that sampled the current batch, and $\delta_t$the per-token gap $\log\pi_{\theta'}(y_t\mid s_t)-\log\mu_{\theta'}(y_t\mid s_t)$ between trainer and sampler at the same weights, in nats for the per-token gap between them, which would be 0 in exact arithmetic. Training maximizes $J(\theta)=\mathbb E_x J(x)$, with $J(x)=\mathbb E_{y\sim\pi_\theta}[R(x,y)]$ for one prompt, but the responses are drawn from $\mu_{\theta'}$, so the unbiased gradient weights each response by its sequence ratio,$$\begin{aligned}&\nabla_\theta J(x)=\mathbb E_{y\sim\mu_{\theta'}}\Big[\underbrace{\frac{\pi_\theta(y\mid x)}{\mu_{\theta'}(y\mid x)}}_{\text{sequence weight}}\\&\qquad\times\nabla_\theta\log\pi_\theta(y\mid x)\,\hat A\Big],\end{aligned}$$where $\hat A$the advantage: the reward $R(x,y)$ minus a baseline that does not depend on $y$, such as the mean reward of the other responses sampled for $x$ is the advantage; the paper calls this estimator PG-Seq-IS. Leaving the weight out biases the gradient whenever $\mu\ne\pi$. The weight is a product of one ratio per token, so its variance grows with length; truncation caps it at a constant $C$the cap on importance weights in the truncated and masked corrections; $C=3$ in the paper's tests, and masking drops responses whose weight exceeds $C$. The other methods start from GRPOgroup relative policy optimization: a clipped per-token policy-gradient objective in which each response's advantage is computed from the rewards of the other responses sampled for the same prompt in its Dr. GRPOthe form of GRPO with no division by the group's reward standard deviation and with each response's summed token losses divided by a constant form, whose per-token ratio $r'_t=\pi_\theta(y_t\mid s_t)/\pi_{\theta'}(y_t\mid s_t)$ divides by the trainer's own probability at $\theta'$. GRPO-Token-TIS multiplies token $t$'s term by $\min(w_t,C)$, with $w_t$the per-token correction weight $\pi_{\theta'}(y_t\mid s_t)/\mu_{\theta'}(y_t\mid s_t)=e^{\delta_t}$; its product over the response is $w(y)$ the correction weight (Yao et al.), and GRPO-Seq-MIS multiplies every term of the response by $w(y)\,\mathbf 1\{w(y)\le C\}$ (Liu et al.). Both need an extra trainer forward pass at $\theta'$, about 25% of the training cost by the authors' count, which PG-Seq-IS avoids: the sampler records $\mu_{\theta'}$, and $\pi_\theta$ comes from the training forward pass. GSPOgroup sequence policy optimization: a GRPO variant that clips a length-normalized ratio of whole-response probabilities instead of per-token ratios; it does not use the sampler's probabilities is also tested.

Main findings (all empirical). (a) Offline, with DeepSeek-R1-Distill-Qwen-1.5B, FP16 in both engines cuts the estimated KL divergence between their response distributions from 7.64 to 0.32 nats, while the sampler's own accuracy is the same in BF16, FP16 and 32-bit (FP32) arithmetic to within about 2.5 points. (b) On a sanity testRL on the 1,460 MATH problems that the starting model solves in 20% to 80% of 40 samples; a sound algorithm should drive training accuracy to nearly 100%, and passing means converging above about 95% with that model, no BF16 method matches FP16: GRPO and GRPO-Token-TIS collapse, and GRPO-Seq-MIS stays stable but reaches only 95% training accuracy and 34% on AIME 2024, against 99% and 39% for PG-Seq-IS in FP16. (c) In FP16 all five algorithms perform almost the same. (d) With a BF16 trainer, raising only the sampler's precision helps, and a 32-bit sampler is fully stable but generates about three times more slowly; FP16 in both engines is stable at full speed. (e) FP16 is also more stable or learns faster for a 30B mixture-of-expertsa model in which each token is processed by a few expert sub-networks chosen by a learned router (top-$k$ selection) out of many model, for LoRAlow-rank adaptation: fine-tuning only a low-rank update $BA$ added to each weight matrix, with the original weights frozen training, for a 14B dense model trained with DAPO's recipe, and for OctoThinker-3B, a model from the Llama family.

Consequences. When RL fine-tuning is unstable, switch the sampler and the trainer to FP16, with the dynamic loss scaling that PyTorch, DeepSpeed and Megatron already provide, before adding importance-sampling corrections. With a small mismatch the plain unbiased estimator works, the extra forward pass of the GRPO-based corrections can be dropped, and the choice among corrections matters little. The mismatch on sampled tokens, such as the mean and extremes of $\pi_{\theta'}-\mu_{\theta'}$, is worth tracking, since in these runs it grew before every collapse. Reweighting can remove the bias of the gradient but leaves the weights tuned for the trainer's probabilities; only making the engines agree helps the model that is deployed. FP16 shrinks the mismatch without removing it, and later work went further: a sampler that matches the trainer bit for bit, which showed that the mismatch alone can collapse training (Zhong et al., 2026), and a trust region measured in total variation from the sampler's probabilities (Qi et al., 2026).

Limits. The evidence is training curves, one run per configuration with no repeated seeds reported, and the most detailed comparisons use a 1.5B model on a problem set filtered so that full accuracy is reachable. The largest models are a 30B mixture-of-experts model and a 14B dense one; the authors expect FP16's range to need care with overflow in much larger models, and the move toward 8-bit training for speed goes the other way. The link from rounding to collapse is shown by changing the precision; the paper does not trace which operations contribute most to the gap. How small the remaining FP16 mismatch must be to be harmless, and whether it stays harmless over much longer runs, is not measured.