Tricks or traps in RL for reasoning

Context. RL recipes for reasoning built on GRPOgroup relative policy optimization: PPO-style clipped updates without a value model, where each response's advantage is its reward standardized within the group of responses to the same prompt stack many small techniques, and published recipes disagree about them. GRPO standardizes rewards within each prompt's group while REINFORCE++a critic-free RL method for language models that standardizes advantages over the whole batch standardizes over the batch; GRPO divides by the standard deviation while Dr. GRPOa GRPO variant that removes the division by each answer's length and by the group's standard deviation, arguing that both bias the gradient argues against it; GRPO averages the loss within each response while DAPOa GRPO variant that raises the upper limit of the ratio clip, averages the loss over all tokens, drops groups with all-equal rewards, and softly penalizes overlong responses averages over all tokens of the batch. Each recipe was tested on its own models, data and settings, so the disagreements may come from the settings. Liu et al. rerun the common techniques one at a time in one framework and on one baseline, with base and post-trained Qwen3 models at 4B and 8B and training sets of three difficulty levels. Most techniques help in some settings and hurt in others, and three properties of the setting decide which: whether the starting model is a base model or an already post-trained one, how often all responses to a prompt get the same reward, and how much of the length limit the reasoning needs. A recipe should be chosen by checking these, and a short one chosen this way (two techniques on the plain PPO loss) trained base models better than GRPO and DAPO. The study extends the argument behind Dr. GRPO, that dividing by the group's standard deviation reweights prompts by difficulty, into an account of when each technique matters.

Dividing advantages by a standard deviation of the rewards enlarges the updates more the more uniform the rewards are, which on easy data they become, and the run destabilizes; with the mean taken per group and the standard deviation over the whole batch, the scale stays bounded (Steps 2 to 4). Raising the upper clipping threshold$\epsilon_{\rm high}$ in PPO's clip of each token's probability ratio to $[1-\epsilon_{\rm low},1+\epsilon_{\rm high}]$; a token whose ratio passes it with positive advantage stops contributing gradient slows the collapse of entropythe average entropy of the policy's next-token distribution over generated tokens; its collapse means the policy has become nearly deterministic and stops exploring in post-trained models and improves their accuracy, while in base models, where the clip is rarely active, it changes little or hurts (Steps 5 and 6). Averaging the loss over tokens helps base models and loses to averaging within responses on post-trained ones (Step 7). Leaving truncated responses out of the loss helps when the length limit is short, where many of them are valid reasoning stopped early, and does little when it is long, where they are mostly repetitive text that never ends (Step 8). Lite PPOthe clipped PPO objective without a value model, with advantages $(r_k-\text{group mean})/\text{batch standard deviation}$ and the loss averaged over all tokens of the batch, which keeps only the batch-scale standardization and token averaging, trains base models more stably than GRPO and DAPO (Step 9).

Setting. All runs use the ROLL framework and one baseline: the PPO clipped losseach token's loss is $-\min(\rho_t\hat A,\ \mathrm{clip}(\rho_t,1-\epsilon_{\rm low},1+\epsilon_{\rm high})\hat A)$, with $\rho_t$ the token's probability ratio between the current policy and the one that sampled it and $\hat A$ the response's advantage with no value model and advantages computed from the rewards alone, as in REINFORCE; 128 prompts with 8 responses each per step, responses of up to 8,192 tokens, learning rate $10^{-6}$, and sampling at temperature 0.99 (top-$p$ 0.99, top-$k$ 100). The reward is 1 for a correct final answer and 0 otherwise. The models are Qwen3-4B-Base and Qwen3-8B-Base, which are pretrained only, and Qwen3-4B and Qwen3-8B, which the paper calls alignedthe released post-trained Qwen3 models, which already reason at length; the paper's term, used here as post-trained. There are three training sets of 5,000 problems each: easy (from SimpleRL-Zoo's easy split, GSM8K and MATH level 1), medium (the easiest 5,000 of DeepMath-103K) and hard (DeepMath-103K sampled in proportion to difficulty), all with most True/False questions removed, since models often reach the right Boolean through wrong reasoning. Test accuracy is averaged over MATH-500, OlympiadBench, Minerva Math, AIME 2024 and 2025 and AMC 2023. For a prompt with $K$the number of responses per prompt, 8 responses and rewards $r_1,\dots,r_K$, the standardizations differ in where the mean and the standard deviation are taken: $$\begin{aligned}\hat A^{\rm group}_k&=\frac{r_k-\mathrm{mean}_{\rm group}}{\mathrm{std}_{\rm group}},\\ \hat A^{\rm batch}_k&=\frac{r_k-\mathrm{mean}_{\rm batch}}{\mathrm{std}_{\rm batch}},\\ \hat A^{\rm lite}_k&=\frac{r_k-\mathrm{mean}_{\rm group}}{\mathrm{std}_{\rm batch}},\end{aligned}$$ with group statistics over the prompt's $K$ rewards and batch statistics over all rewards in the batch. Sequence-level aggregationaverage each response's token losses, then average over responses, so every response has the same total weight and token-level aggregationsum all token losses in the batch and divide by the total number of tokens, so every token has the same weight differ in how token losses are combined. Clip-higherclipping the ratio to $[1-\epsilon_{\rm low},1+\epsilon_{\rm high}]$ with $\epsilon_{\rm high}\gt\epsilon_{\rm low}$; the default is 0.2 on both sides, DAPO's upper value 0.28 raises only the upper threshold, and overlong filteringleaving responses truncated at the length limit out of the loss instead of scoring them as wrong changes how truncated responses count.

Main findings (all empirical). Post-trained models start far higher than base models and gain only about 2 points of accuracy from RL in this setup. With 0/1 rewards, standardizing within groups was more stable than standardizing over the batch, whose runs sometimes collapsed when a few outlying responses dominated a skewed batch. On easy data the batch's reward standard deviation fell from about 0.48 to below 0.25, and runs that divided by it collapsed while runs that only subtracted the mean kept improving; on hard data it stayed near 0.5 and both behaved alike. With the mean taken per group, dividing by the batch's standard deviation beat dividing by the group's on most benchmarks for both base models, on easy and hard data. Raising $\epsilon_{\rm high}$ from 0.2 to 0.28 slowed the fall of entropy and raised accuracy for post-trained models and had little effect on base models or slightly lowered their accuracy; Qwen3-4B kept improving up to 0.32, the largest value tried, and Qwen3-8B did best at 0.28. Token-level aggregation beat sequence-level for Qwen3-8B-Base and lost to it for Qwen3-8B. Overlong filtering helped Qwen3-8B-Base clearly at a limit of 8K tokens and little at 20K. Lite PPO beat GRPO and DAPO on both base models, most clearly on Qwen3-4B-Base, where GRPO and DAPO collapsed after their peak.

Consequences and limits. The guidance reads as a short decision list. From a base model, standardize with the group mean and the batch standard deviation, average the loss over tokens, and keep the symmetric clip, which is Lite PPO. From a post-trained reasoning model, raise the upper clipping threshold (0.28, higher for a 4B model), average within responses, and expect small gains. When most groups are all right or all wrong, as on data that is easy for the model, avoid dividing by a group's standard deviation: drop the division, or divide by the batch's with the group mean, as Lite PPO does (though the runs that collapsed on easy data also divided by the batch's standard deviation, and which mean they subtracted is unclear; Technical notes). When the length limit is short for the reasoning the task needs, filter truncated responses out of the loss; when it is long, filtering matters less but keeps the model from drifting into repetition that never ends. The evidence has limits, some of which the authors state: one model family (Qwen3), one smoothed curve per setting with no repeated seeds, comparisons read off curves on six benchmarks of which two have 30 problems, and Lite PPO tested only on base models against GRPO and DAPO in their standard forms. Base and post-trained models differ in several ways at once (accuracy, entropy, response length), so a finding labeled by model type may follow any of them.