How Off-Policy Can GRPO Be?

Context. GRPOgroup relative policy optimization: PPO without a value model, in which every token of an answer gets the answer's reward minus its group's mean reward, divided by the group's standard deviation, as advantage for RL with verifiable rewardsreinforcement learning in which a program checks each answer and returns the reward, here 1 for a correct final answer and 0 otherwise is usually run close to on-policy: the current model samples a batch of answers, a few gradient steps are taken on it (four is a common choice), and then the model samples again. Each refresh moves the new weights from the training engine to the inference engine that generates answers, and to switch quickly between the two, systems keep the trainer's parameters and optimizer state on the GPUs during generation, which leaves less memory for generating. Training is already somewhat off-policy in practice, through this reuse of each batch, through asynchronous pipelines and through numerical differences between the two engines. Tian et al. (2026) ask how much further the data can lag behind the policy, and push GRPO to over a thousand updates per batch of samples.

Stale samples still carry useful gradient after more than a thousand updates, and what goes wrong at that staleness is specific and can be filtered out. With GRPO's usual clip interval, more and more stale tokens fall outside it and stop contributing, so learning stalls. With a much wider interval, training learns fast at first and then collapses, because the clip's lower bound was what stopped the update from pushing down tokens of below-average answers: without it, the same sampled answers are suppressed again and again, and their tokens become far less likely than when they were sampled. The damage comes from the tokens that follow such a token (one that has become far less likely), which the policy is still trained on although the prefix they follow has become very unlikely under it. Dropping any below-average answer that contains such a token, together with the wide interval, lets GRPO train in four stages of 1024 updates each. On four of five models this matched or beat ordinary GRPO, with end-to-end training time 1.53 times shorter on average over the five models.

Setting. For a question $q$, the sampling policy $\pi_{\rm old}$the policy that generated the current data, frozen while it is used; $\pi_\theta$ is the policy being trained (the paper writes $\beta$ for the sampling policy) draws $G$the number of answers sampled per question, 8 in the experiments answers $o_1,\dots,o_G$; write $o_{i,t}$ for token $t$ of $o_i$, $|o_i|$ for its length, and $\pi_\theta(o_{i,t}\mid\cdot)$ for the probability of $o_{i,t}$ given $q$ and $o_{i,\lt t}$. Each answer gets a reward $R_i\in\{0,1\}$ and every one of its tokens the advantage $\hat A_i$$\hat A_i=(R_i-\bar R)/\operatorname{std}(R)$, with $\bar R$ and $\operatorname{std}(R)$ the mean and standard deviation of the rewards in the group of $q$; negative for answers below the group's average. The staleness $\mu$the number of optimizer updates made on one batch of sampled answers: the batch's number of questions divided by the number per update; $\mu=1$ is fully on-policy counts the updates made before the data are refreshed; ordinary GRPO uses $\mu=4$, and μ-GRPO uses $\mu=1024$ with 32 questions per update. With $\rho_{i,t}$$\rho_{i,t}=\pi_\theta(o_{i,t}\mid\cdot)/\pi_{\rm old}(o_{i,t}\mid\cdot)$, the probability ratio of token $t$ of $o_i$, computed from the log-probabilities stored at sampling time the ratio of token $t$, $\rho_{i,t}=\pi_\theta(o_{i,t}\mid\cdot)/\pi_{\rm old}(o_{i,t}\mid\cdot)$; GRPO maximizes the per-token clipped objective $\min\big(\rho_{i,t}\hat A_i,\operatorname{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_i\big)$, averaged over the tokens of each answer and then over answers, with $\varepsilon=0.2$GRPO's clip range, giving the interval [0.8, 1.2] and no KL term. μ-GRPO widens the interval to $[0,5]$ and adds a rule the paper calls the negative-advantage veto. A token is a triggera token of an answer with negative advantage whose ratio has fallen below $\tau_c$: the current policy finds it at least $10^4$ times less likely than the sampling policy did if $\hat A_i\lt0$ and $\rho_{i,t}\lt\tau_c$, with $\tau_c=10^{-4}$the trigger threshold on a token's ratio; let $m_i$the veto mask of $o_i$: 0 if $o_i$ has negative advantage and contains a trigger token at the current update, 1 otherwise; recomputed at every update and not differentiated be 0 for an answer with a trigger and 1 otherwise. Over a minibatch $B$ of answers, μ-GRPO maximizes$$\begin{aligned}&\frac1{|B|}\sum_{i\in B}\frac{m_i}{|o_i|}\sum_{t=1}^{|o_i|}\\&\quad\min\big(\rho_{i,t}\hat A_i,\operatorname{clip}(\rho_{i,t},0,5)\hat A_i\big).\end{aligned}$$Training runs in stages: $\pi_{\rm old}$ is frozen, generates answers to 32,768 questions (1024 updates times 32) with their log-probabilities, is released from memory, and the trainer makes 1024 updates on that data before the next stage begins from the updated policy.

Main findings (empirical apart from an appendix bound). (a) At $\mu=1024$, GRPO with its usual clip loses accuracy: on Qwen2.5-Math-7B the average over five math benchmarksAMC 2023 and 2024, AIME 2024 and 2025, and MATH500, scored by pass@1 averaged over 16 samples per problem for AIME and 4 for the others falls from 47.3% at $\mu=4$ to 41.8%. μ-GRPO at $\mu=1024$ gets 48.0%. (b) Against ordinary GRPO at $\mu=4$, μ-GRPO matches or improves the average on four of five models (1.7B to 8B), with the largest gain on DeepSeek-7Bthe paper's name for DeepSeek-R1-Distill-Qwen-7B, Qwen2.5-Math-7B fine-tuned on reasoning traces of DeepSeek-R1 (68.6% against 66.1%) and a loss on Qwen3-8B-Base (48.2% against 50.3%), and it makes generation time 1.82 times and end-to-end training time 1.53 times shorter, each the mean over the five models of GRPO's time divided by μ-GRPO's. (c) Relaxing the clip without the veto collapses mid-training; vetoing only the trigger tokens still collapses, while vetoing the tokens after the first trigger is stable. (d) The veto removes few tokens, on average over stages 0.17% to 4.03% of tokens depending on the model, and the result is similar for the three veto scopes that include the post-trigger tokens.

Consequences. For GRPO-style training on verifiable rewards, rollouts can be generated in a few large batches and reused for about a thousand updates, which removes most weight synchronizations and lets generation use all the GPU memory, provided the clip interval is wide and below-average answers containing a token whose ratio has fallen below $10^{-4}$ are dropped. With stale data, ordinary clipping is the wrong safeguard in two ways: it stops too many useful tokens, and it is the only thing limiting how far below-average answers are pushed down, so removing it requires another limit. The rule for the veto follows from the cause: once a sampled answer contains a token the current policy has nearly ruled out, the rest of that answer follows a prefix the policy would almost never produce, so its later tokens should not be trained on.

Limits. No seeds or error bars are reported, and the reported checkpoint is the best of those evaluated every 200 updates, chosen by its average on the same five test benchmarks. The models are 1.7B to 8B with 4k to 8k-token contexts on four GPUs, trained on one math dataset; whether a thousand updates of staleness is safe for larger models or longer answers is not tested. μ-GRPO is behind GRPO on the largest model, Qwen3-8B-Base, and within two points of M2POa method for stale data (Zheng et al., 2025) that masks high-variance tokens until a batch-level second moment of the importance weights is below a threshold, 0.04 here, the closest earlier method, on every model, while faster than it (1.49 times on average). The 2.2-times speedup in the paper's opening figure is a time-to-accuracy ratio on DeepSeek-7B: the wall-clock time GRPO took to reach its best accuracy divided by the time μ-GRPO took to reach the same accuracy. That model's end-to-end training time fell only 1.21 times (61.9 to 51.3 hours, my computation from the paper's Table 1), and the abstract's "around 2×" speedup in training time is above the table's five-model average of 1.53.