DAPO and Dr. GRPO

Context. After DeepSeek-R1, open attempts to reproduce R1-ZeroDeepSeek's model trained with answer-checking rewards by reinforcement learning directly from a pretrained base model, with no supervised stage ran GRPOgroup relative policy optimization: a policy-gradient method that samples a group of answers to each question, gives each answer its reward minus the group mean, divided by the group's standard deviation, as its advantage, and clips each update so the policy moves only a little with answer-checking rewards on open base modelspretrained language models that have had no instruction tuning or other fine-tuning and ran into trouble. On Qwen2.5-32B a straightforward GRPO run reached 30% accuracy on AIME 2024the 30 problems of the 2024 American Invitational Mathematics Examination, each with an integer answer, against 47% reported by DeepSeek for the same base model, with the policy's entropythe average entropy of the policy's next-token distributions on its own samples, a measure of how much its sampling still varies collapsing early; other runs produced ever longer answers. DAPO (Yu et al., 2025) and Dr. GRPO (Liu et al., 2025) each trace such failures to specific terms in GRPO's objective and change them. Both drop the KL penaltya term proportional to the Kullback-Leibler divergence of the trained policy from a reference model, subtracted from the objective to keep the policy near that model and both change how per-token losses are averaged; DAPO also raises the upper clip bound, discards groups whose answers all get the same reward and softens the penalty on truncated answers, and Dr. GRPO also drops the division by the group's standard deviation. Liu et al. also look at the base models, and find that much of what R1-Zero-style training is credited with is already there before RL.

GRPO's two normalizations reweight its gradient in ways the reward does not ask for. Averaging each answer's token losses over its own length gives every answer the same total weight whatever its length, so a long wrong answer is penalized less per token than a short one, and length can grow while accuracy does not. Dividing by the group's reward standard deviation gives more weight to questions that nearly every answer, or nearly none, gets right. The symmetric clip limits how fast unlikely tokens can gain probability, which speeds entropy collapse, and groups whose answers all get the same reward contribute no gradient. Each fix removes one of these effects, and with both normalizations removed GRPO is RLOOREINFORCE leave-one-out: the policy gradient in which each answer's baseline is the mean reward of the other answers sampled for the same question, the leave-one-out policy gradient, up to a constant step size.

Setting. For a question $q$, sample $G$the number of answers sampled per question: 16 in DAPO, 8 in Liu et al. answers $o_1,\dots,o_G$ from the sampling policy $\pi_{\rm old}$the policy as it was when the group was sampled; the policy being trained is $\pi_\theta$ and score each with a verifiable reward $R_i$: 1 for a correct final answer and 0 otherwise in Liu et al., $+1$ or $-1$ in DAPO. Write $|o_i|$ for the length of $o_i$ in tokens, $\bar R$ and $\operatorname{std}(R)$ for the mean and standard deviation of the group's rewards, and $\rho_{i,t}$the probability ratio $\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})/\pi_{\rm old}(o_{i,t}\mid q,o_{i,\lt t})$ of token $t$ of $o_i$$=\pi_\theta(o_{i,t}\mid q,o_{i,\lt t})/\pi_{\rm old}(o_{i,t}\mid q,o_{i,\lt t})$, the probability ratio of token $t$ of $o_i$. GRPO gives every token of $o_i$ the advantage $\hat A_i=(R_i-\bar R)/\operatorname{std}(R)$ and maximizes$$\begin{aligned}\mathbb E\Big[&\frac1G\sum_i\underbrace{\frac1{|o_i|}}_{\text{length}}\sum_t\min\big(\rho_{i,t}\hat A_i,\\&\operatorname{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_i\big)\Big],\end{aligned}$$here with no KL term, where $\varepsilon$the clip range, 0.2 by default: for a token with positive advantage, the term stops contributing gradient once $\rho_{i,t}\gt1+\varepsilon$, and for negative advantage once $\rho_{i,t}\lt1-\varepsilon$ is the clip range. Dr. GRPO replaces $\hat A_i$ by $R_i-\bar R$ and divides each answer's summed token losses by one constant for all answers, such as the generation budget, instead of by $|o_i|$. DAPO keeps $\hat A_i$, divides the summed token losses of the group by its total length $\sum_i|o_i|$, clips at $1-\varepsilon_{\rm low}$ and $1+\varepsilon_{\rm high}$ with $\varepsilon_{\rm high}\gt\varepsilon_{\rm low}$separate lower and upper clip ranges, 0.2 for the lower and 0.28 for the upper in DAPO, and trains only on groups with $0\lt\#\{\text{correct}\}\lt G$why: if every answer in a group gets the same reward, every $R_i-\bar R$ is 0 and the group adds nothing to the gradient except a smaller average (Step 6).

Main findings (all empirical except the bias computation). (a) DAPO, on Qwen2.5-32B with 17k math problems whose answers were rewritten as integers, reaches 50% on AIME 2024 (accuracy averaged over 32 samples per problem) against 30% for plain GRPO and 47% for DeepSeek's R1-Zero recipe on the same base model, in half of DeepSeek's training steps. Adding the techniques one at a time gives 30, 36 (masking truncated answers), 38 (higher upper clip), 41 (soft length penalty), 42 (token-level averaging) and 50 (dynamic sampling). (b) Liu et al. show that dividing by $|o_i|$ favors short correct answers and long wrong ones, and dividing by $\operatorname{std}(R)$ favors questions solved by nearly all or nearly no answers. On a 1.5B model, GRPO's wrong answers keep lengthening after reward levels off, while Dr. GRPO keeps them shorter at equal or better accuracy, consistently over three seeds. (c) Their minimal recipe, Dr. GRPO on Qwen2.5-Math-7B with MATHa dataset of competition math problems in five difficulty levels, scored by exact final answer problems of levels 3 to 5 and the Qwen-Math template for 27 hours on eight A100 GPUs, scores 43.3% on AIME 2024. (d) Base models matter: the Qwen2.5-Math models answer best with no prompt templatethe fixed text wrapped around each question, such as R1's instruction to reason inside think tags before answering at all, and DeepSeek-V3-Base already writes self-reflection phrases such as "wait" and "aha" before RL.

Consequences. For RL with verifiable rewards these papers propose several defaults: no KL penalty, token losses summed and divided by a constant or by the group's token count, no division by the group's standard deviation, an upper clip range a little above the lower one, groups with all-equal rewards filtered out, and truncated answers masked or penalized gently. Growing response length is no evidence of learned reasoning by itself; the length of wrong answers, and accuracy at a fixed length, say more. Before crediting RL with a gain, score the base model with the template that suits it, since a mismatched template can suppress ability that RL then appears to create.

Limits. DAPO's ablation is one cumulative sequence on one model, each technique added once in one order, and AIME 2024 has 30 problems, so a point of accuracy is less than a third of one problem; averaging 32 samples per problem reduces sampling noise but not that granularity. DAPO also reports that training reward correlates little with validation accuracy. Liu et al.'s RL experiments use 1.5B to 7B models with 3,000-token budgets, so the size of the length effect at R1's scale and lengths is not measured. DAPO's token-level averaging still divides by each group's total length, so a group of long answers has less weight per token than a group of short ones, which Dr. GRPO's constant avoids; neither paper compares the two directly. Both papers drop the KL penalty for rule-based rewards; with a learned reward the argument for dropping it does not apply.