The trust region in LLM reinforcement learning

Context. PPOproximal policy optimization: a policy-gradient method that takes several gradient steps on each sampled batch and clips each sampled token's probability ratio so the policy moves only a little per batch, the standard algorithm for reinforcement-learning (RL) fine-tuning of language models, descends from TRPOtrust region policy optimization: maximize a first-order estimate of the improvement subject to a bound on the KL divergence between the new policy and the one that collected the data, which keeps each new policy within a set divergence of the policy that collected the data, a constraint called a trust regionthe set of policies close enough to the data-collecting policy, in KL divergence or total variation, that a first-order estimate of the improvement can be trusted. PPO replaces the constraint by clipping each sampled token's probability ratio. In language-model RL the data are off-policy for two reasons: several gradient steps are taken on each batch, and the training-inference mismatchthe difference between the token probabilities that the inference engine (the sampler) and the training engine (the trainer) assign to the same response under the same weights makes the trainer's probabilities differ from the sampler's even before the first step. Qi et al. (2026) rederive the trust-region bound for responses of bounded length with one reward at the end, argue that ratio clipping measures distance badly over the long-tailed vocabularies of language models, and replace it with a direct estimate of the divergence.

The total variation$D_{\rm TV}(p,q)=\tfrac12\sum_a|p(a)-q(a)|$: the largest difference between the probabilities two distributions give the same event, between 0 and 1 between the sampler's and the trained policy's next-token distributions is half the expected $|r-1|$ over tokens the sampler draws, with $r$ the ratio of their probabilities. PPO's test that $|r-1|$ of the one sampled token stays within the clip range therefore tests a one-sample estimate of twice this distance, with an error that depends on the token's probability: raising a token from $10^{-4}$ to $10^{-2}$ gives ratio 100 and is clipped though it moves less than 0.01 of probability, while lowering one from 0.99 to 0.80 gives ratio 0.81 and passes a clip range of 0.2 though it moves 0.19. DPPOdivergence proximal policy optimization: PPO with the ratio test replaced by a test on an estimate of the divergence between the two policies' next-token distributions keeps PPO's form and blocks a token's update only when an estimate of the divergence at that position exceeds a threshold and the update moves further away. Its cheap binary estimate, the change in the sampled token's probability, equals $\mu\,|r-1|$, the ratio's deviation weighted by the token's probability $\mu$ under the sampler. Measured from the sampler's probabilities, DPPO trains stably where methods with no trust region, or with one measured from the trainer's recomputation, collapse, and it learns faster than GRPOgroup relative policy optimization: PPO with each response's advantage computed from the rewards of the other responses sampled for the same prompt with Clip-HigherDAPO's change to PPO's clip: a larger upper clip range (0.28) than lower (0.2), so tokens with positive advantage can gain probability faster.

Setting. A prompt gets a response $y=(y_1,\dots,y_{|y|})$ of at most $T$the maximum response length in tokens, the horizon of the finite-horizon problem tokens and a reward $R(y)$ at the end, and $s_t$the prefix $(x,y_1,\dots,y_{t-1})$ before token $t$, with $x$ the prompt is the prefix before token $t$. Write $\mu_{\theta'}$the sampler's policy at the weights $\theta'$ that generated the batch for the sampler's policy at the weights that generated the batch, $\pi_{\theta'}$the trainer's recomputation of the policy at the same weights $\theta'$ for the trainer's recomputation of it, and $\pi_\theta$the trainer's policy at the weights $\theta$ being updated for the policy being trained. The theory holds for any two policies; write $\mu$ for the one that generated the data, $\pi$ for the one being trained, $J(\pi)=\mathbb E_{y\sim\pi}[R(y)]$, and $r_t$${}=\pi(y_t\mid s_t)/\mu(y_t\mid s_t)$ for the ratio at token $t$. The distance at a prefix $s$ is $D_{\rm TV}(s)$the total variation between $\mu(\cdot\mid s)$ and $\pi(\cdot\mid s)$, the two policies' next-token distributions after prefix $s$, and with $r(a)=\pi(a\mid s)/\mu(a\mid s)$ for a token $a$,$$\begin{aligned}D_{\rm TV}(s)&=\tfrac12\textstyle\sum_a|\mu(a\mid s)-\pi(a\mid s)|\\&=\tfrac12\,\mathbb E_{a\sim\mu(\cdot\mid s)}|r(a)-1|.\end{aligned}$$PPO, with GRPO as a variant of it, maximizes$$\begin{aligned}&\mathbb E_{y\sim\mu}\Big[\sum_t\min\big(r_t\hat A,\\&\qquad\operatorname{clip}(r_t,1-\varepsilon,1+\varepsilon)\hat A\big)\Big],\end{aligned}$$with $\hat A$the advantage $R(y)-\frac1G\sum_{i=1}^GR(y_i)$: the response's reward minus the mean reward of the $G$ responses sampled for the same prompt, shared by all its tokens the advantage and $\varepsilon$the clip range; with separate lower and upper ranges $\varepsilon_{\rm low}=0.2$ and $\varepsilon_{\rm high}=0.28$ (0.27 in the large runs) the interval is $[1-\varepsilon_{\rm low},1+\varepsilon_{\rm high}]$ the clip range. Clipping removes a token's gradient once $r_t$ leaves the interval in the direction its advantage pushes.

Main result (theory). The first theorem asks how much the expected reward changes when the policy moves from $\mu$ to $\pi$, written with expectations under $\mu$ only, the ones a batch of samples can estimate. The answer is a first-order term, the objective PPO optimizes before clipping, minus an error that comes from changes at later tokens; the second theorem bounds the error by the per-token distance. For responses of at most $T$ tokens with no discountingwhy: the classical trust-region bound has a factor $1/(1-\gamma)^2$ that is infinite with no discounting ($\gamma=1$); with responses of at most $T$ tokens, $T$ takes its place, any two policies satisfy$$J(\pi)-J(\mu)=\underbrace{L'_\mu(\pi)}_{\text{first order}}-\underbrace{\Delta(\mu,\pi)}_{\text{error}},$$where the sums and the product run over the tokens of $y$:$$L'_\mu(\pi)=\mathbb E_{y\sim\mu}\Big[R(y)\sum_t(r_t-1)\Big],$$$$\begin{aligned}\Delta(\mu,\pi)=\mathbb E_{y\sim\mu}\Big[&R(y)\sum_t(r_t-1)\\&\times\Big(1-\prod_{j\gt t}r_j\Big)\Big].\end{aligned}$$Let $\xi=\max_y|R(y)|$why: Step 2 bounds the error term by taking $|R(y)|\le\xi$ out of the expectation, so the bounds need a bounded reward, let $D^{\max}_{\rm TV}$the largest per-prefix distance, $\max_sD_{\rm TV}(s)$ over all prefixes $s$ be the largest per-prefix distance, and let $\bar D_{\rm TV}$the summed per-token distance along a sampled response, averaged over responses: $\mathbb E_{y\sim\mu}\big[\sum_tD_{\rm TV}(s_t)\big]$ be the summed per-token distance along a response, averaged over responses drawn from $\mu$. Then both bounds hold:$$\begin{aligned}&J(\pi)-J(\mu)\ge L'_\mu(\pi)\\&\qquad-\underbrace{2\xi T(T-1)\big(D^{\max}_{\rm TV}\big)^2}_{\text{worst prefix}},\end{aligned}$$$$\begin{aligned}&J(\pi)-J(\mu)\ge L'_\mu(\pi)\\&\qquad-\underbrace{4\xi\,\bar D_{\rm TV}}_{\text{along the response}}.\end{aligned}$$The first has the form of TRPO's bound, with $T$ in place of $1/(1-\gamma)$; the second grows only linearly with length and is what a per-token constraint controls.

DPPO. DPPO maximizes $\mathbb E_{y\sim\mu}\big[\sum_tM_t\,r_t\,\hat A\big]$, with the mask $M_t$ held fixed in the gradient:$$M_t=\begin{cases}0,&\hat A\gt0,\ r_t\gt1,\ D_t\gt\tau,\\0,&\hat A\lt0,\ r_t\lt1,\ D_t\gt\tau,\\1,&\text{otherwise,}\end{cases}$$where $D_t$an estimate of the total variation or KL divergence between $\mu_{\theta'}(\cdot\mid s_t)$ and $\pi_\theta(\cdot\mid s_t)$, the sampler's and the trained policy's next-token distributions estimates the divergence at $s_t$ and $\tau$DPPO's divergence threshold: 0.15 for total variation and 0.05 for KL in the stability tests, 0.2 for total variation in the large runs is a threshold. With $D_t$ replaced by $|r_t-1|$ and $\tau$ by $\varepsilon$ this is PPO's clipped objective, so DPPO keeps PPO's structure, including that it never blocks a move back toward ratio 1, and changes only the test. Computing $D_t$ over the whole vocabulary at every position takes too much memory, so two approximations group the tokens. The binary form compares the sampled token against the rest, so its total variation is $|\mu_{\theta'}(y_t\mid s_t)-\pi_\theta(y_t\mid s_t)|$ and its KL divergence is that of two coin flips with these probabilities; the top-K form keeps the $K$the number of most likely tokens kept separately in the top-K approximation, 20 in the paper most likely tokens under the sampler and the sampled token, and merges the rest. Throughout, ratios and divergences are measured from the sampler's probabilities $\mu_{\theta'}$.

Main findings (empirical). (a) In a small stability test run in BF16, methods with no trust region, or with one measured from the trainer's recomputation $\pi_{\theta'}$, let the mismatch grow and collapse, while DPPO keeps it low and reaches near-perfect training reward. (b) Blocking only the updates on wrong responses that lower the sampled token's probability by at least 0.5, at most 0.5% of updates, stabilizes the unconstrained policy gradient. (c) Truncating the importance ratio makes collapse come sooner. (d) Removing both ends of the clip for tokens with probability below 0.1 speeds learning. (e) On 30B mixture-of-expertsa model in which each token is processed by a few expert sub-networks, chosen by a learned router out of many and 8B dense Qwen3 models, DPPO learns faster and ends higher on AIME 2024 and 2025 than GRPO with Clip-Higher and than CISPOa policy-gradient method that keeps every token's gradient and multiplies it by the importance ratio truncated at a cap $C$; it masks no token, and with a learned reward it beats GRPO too. (f) The binary approximation does as well as the top-20 one.

Consequences. For PPO-style RL on language models, measure the trust region from the probabilities the sampler recorded; this also removes the recomputation pass, about 25% of training cost by the authors' estimate. Replacing the ratio test by a mask on the change in the sampled token's probability is a few lines of code, and the threshold mattered little between 0.1 and 0.2 for total variation. Keep a trust region even at learning rates of $10^{-6}$. Watch updates on wrong responses that push a likely token far down, and avoid truncating the whole importance ratio, which biases the gradient against rare tokens. FP16 in both engines (Qi et al., 2025) and a sampler that matches the trainer exactly (Zhong et al., 2026) reduce the mismatch itself; DPPO works with the BF16 mismatch present and gains further when it is reduced, as with routing replayR3: recording which experts the sampler's router chose for each token of a mixture-of-experts model and reusing those choices in the trainer, which removes one source of mismatch for mixture-of-experts models.

Limits. The comparisons are single training runs read off curves, scored on AIME sets of 30 problems each, with models up to 30B parameters, and the baselines use settings from other papers. The causal claims of the stability study, that a few large negative updates drive collapse and that sharply lowering likely tokens damages the model, rest on which masks stabilize one 1.5B model on one filtered set. The bounds justify the form of the constraint but give no usable guarantee at these lengths (Step 2). Like PPO, DPPO judges each update by the sampled token only, and the binary distance ignores how the rest of the distribution moves.