GRPO and DeepSeek-R1

The group mean replaces the value model as the baseline. PPO computes each token's advantage with GAEgeneralized advantage estimation: $\hat A_t=\sum_{l\ge0}(\gamma\lambda)^l\delta_{t+l}$ with $\delta_t=R_t+\gamma V(s_{t+1})-V(s_t)$, where $R_t$ is the reward at token $t$, $V$ the value model, $\gamma$ a discount and $\lambda\in[0,1]$ trades bias for variance from a value model trained alongside the policy, and in RLHFreinforcement learning from human feedback: RL against a reward model trained on human comparisons of responses the reward model scores only the last token, so the value model must predict the final reward from every prefix. GRPO samples $G$ answers per question and uses their mean reward as the baseline, dividing by their standard deviation; with a reward on the whole answer, every token of $o_i$ gets $\hat A_i$. The group mean $\bar r$ includes $r_i$ itself; a standard identity, not stated in the paper, shows that centering on it equals centering on the leave-one-out mean, up to a constant factor,$$r_i-\bar r=\tfrac{G-1}{G}\Big(r_i-\frac1{G-1}\sum_{j\ne i}r_j\Big).$$The leave-one-out mean does not depend on $o_i$, so subtracting it leaves the expected gradient unchanged (the score $\nabla_\theta\log\pi_\theta(o_i\mid q)$ has mean zero), and the centered estimator has the direction of the policy gradient on average; the division by the group's standard deviation changes how much each question weighs, which Liu et al. (2025) later examined. The KL penalty moves out of the per-token reward, where PPO for RLHF puts it, into the loss, and its per-token estimator has mean $D_{\rm KL}(\pi_\theta\|\pi_{\rm ref})$ when tokens are drawn from $\pi_\theta$. DeepSeekMath also defines a process-supervision variant, in which a process reward modela learned model that scores each intermediate reasoning step, as opposed to only the final answer scores each step, the step scores are normalized across the group, and each token gets the sum of the normalized scores of the steps from it onward.

Sampling from the current policy and penalizing wrong answers both add accuracy. Shao et al. write each method's gradient as an average over a data source of a per-token gradient coefficient times $\nabla_\theta\log\pi_\theta(o_t\mid q,o_{\lt t})$. Supervised fine-tuning uses human answers with coefficient 1. Rejection-sampling fine-tuning (RFT)sampling answers from a fixed model, keeping the correct ones, and fine-tuning on them with ordinary next-token loss samples from the supervised model and gives its correct answers coefficient 1; online RFT does the same with samples from the current policy. GRPO's coefficient is $\hat A_i+\beta(\pi_{\rm ref}/\pi_\theta-1)$, graded by reward and negative for answers below the group mean. On DeepSeekMath-Instruct 1.3B, online RFT pulls ahead of RFT later in training, once the policy has moved away from the supervised model; GRPO beats online RFT; and process supervision beats outcome supervision. On the 7B model, two rounds of iterative RL, which retrain the reward model on the policy's samples (with 10% of the old data replayed) and reset the reference, add more, mostly in the first round. The method comparisons are on one 1.3B model.

RL raised Maj@K at every K and Pass@K only at small K. At temperature 0.7 on GSM8K and MATH, the RL model's Maj@K exceeds the Instruct model's for $K$ from 1 to 64. Its Pass@K is higher at $K=1$, where Pass@1 and Maj@1 are the same number, but the gap is gone by $K=16$, and on GSM8K the Instruct model is slightly ahead at $K=64$ (read off the paper's Figure 7). The authors read this as RL making the output distribution more robust: it makes correct answers that were already among the $K$ samples the most common and adds few problems the model could not already solve. They suggest two reasons specific to their setup, that the RL questions came from the supervised data and that sampling was plain nucleus samplingsampling each token from the smallest set of most likely tokens whose total probability exceeds a threshold. The measurement is for one 7B model with answers capped at 1,024 tokens; it does not settle what longer RL from a base model does, which is what R1 tests.

R1-Zero is GRPO on a base model with a rule-based reward and a fixed template. The policy starts from DeepSeek-V3-Base, a 671B mixture-of-expertsa transformer in which each token passes through only a few of many feed-forward blocks, so only part of the parameters is active per token model, and a template asks it to reason inside <think> tags and answer inside <answer> tags, with no guidance on how to reason. The reward is rule-basedwhy: R1 found that neural reward models, outcome or process, are open to reward hacking during large-scale RL, and retraining them adds compute and complexity, on math, code, STEM multiple-choice and logic questions. Training uses learning rate $3\times10^{-6}$, $\beta=0.001$, sampling temperature 1, $G=16$, and 32 questions per step, so each step trains on 512 answers; each rollout generates 8,192 answers, split into 16 minibatches of 512 and trained on once (the note's reading: one rollout feeds 16 steps). The maximum length is 32,768 tokens until step 8,200 and 65,536 after, and both accuracy and length jump at the switch. The run lasts 10,400 steps, 1.6 epochs over the data, and the reference policy is replaced by the current policy every 400 steps.

Accuracy and response length grow together, and reflective behavior appears during training. AIME 2024 pass@1 rises from 15.6% to 77.9%. The average training response length rises steadily to many thousands of tokens without any length reward. On MATH, the easy levels are solved early and accuracy on the hardest level goes from about 0.55 to 0.90. Counts of reflective words ("wait", "mistake", "verify", "check" and others) rise five- to sevenfold over training, and "wait" in particular is nearly absent early, occasional between steps 4,000 and 7,000, and frequent after 8,000; the paper shows one transcript in which the model stops, writes that this is an "aha moment", and re-derives its answer. All of this is measured on one run. It shows that these behaviors arose without examples of them in training, since R1-Zero has no supervised stage before RL; it does not show they were absent from the base model, and later work found them there (see the limits below).

The R1 pipeline restores readability and general skills, at an early cost in reasoning. R1-Zero mixes English and Chinese and is hard to read, and its RL covered only reasoning. R1 first fine-tunes the base model on thousands of long solutions written in a first-person, readable style (R1-Zero outputs rewritten by annotators and then by a model), runs reasoning RL with a language-consistency rewardthe fraction of words in the chain of thought that are in the target language, added to the reward (which costs a little accuracy in an ablation), then fine-tunes on 800k samples of reasoning and other data, and ends with an RL stage that combines rule rewards with helpfulness and safety reward models, at temperature 0.7. The preference rewards enter only in the last 400 of that stage's 1,700 steps. Along the way AIME 2024 pass@1 is 59.0% after the small first fine-tune, against R1-Zero's 77.9%, and recovers to 79.8% for R1, while AlpacaEval 2.0a benchmark of open-ended instructions on which a strong model judges which of two responses is better; LC is its length-controlled win rate length-controlled win rate goes from R1-Zero's 24.7% to R1's 87.6%. The paper sets the clip range $\varepsilon$ to 10 in the first RL stage and notes that small values remove the gradient of many tokens while large values destabilize training.

PPO nearly matches GRPO when GAE uses $\lambda=1$, not at 0.95. On MATH with DeepSeek-Coder-V2-Lite (16B mixture-of-experts, 2.4B active parameters), PPO with GAE at $\lambda=1$why: with $\lambda=1$, GAE's sum telescopes to the observed return minus $V(s_t)$, so the value model enters only as a baseline and its errors add noise without bias (standard GAE algebra; the paper gives no reason) nearly reaches GRPO, while the common default $\lambda=0.95$ is considerably worse. The paper gives two further reasons to prefer GRPO for long chains of thought. A value model must predict the final reward from a partial answer whose content may later be revised, which is hard. And PPO's KL penalty enters the reward at every token, so the return accumulates it over the whole answer and implicitly penalizes length, while GRPO's KL term sits in the loss and is not summed into a return.

RL from a general base model worked from 32B up, and distillation beat it at 32B. Early attempts with a 7B dense and a 16B mixture-of-experts base model failed to improve AIME: as responses grew longer they became repetitive. Pure RL worked from 32B, 230B and 671B models. The threshold is not sharp: a math-specialized 7B base, Qwen2-Math-7B, trained with about 10,000 RL steps reached 22.3% on AIME 2024, against 7.9% for its instruct version. Qwen2.5-32B-Base trained with the same RL for over 10k steps reaches 47.0% on AIME 2024, against 72.6% for the same base model fine-tuned on 800k R1 outputs, and the distilledfine-tuned with ordinary supervised loss on outputs generated by a stronger model 1.5B model scores 28.9% where GPT-4o scores 9.3%. Two methods gave no net benefit at R1's scale: a process reward model (steps are hard to define and to label, and a learned model gets hacked) and Monte Carlo tree searcha search that grows a tree of partial solutions, choosing which branch to extend with a learned value model over reasoning steps (the token-level search space is too large and the value model too hard to train).

Limits. The evidence is mostly single training runs read off benchmark curves, and the reasoning behaviors are measured by counting words. Later analyses found self-reflection words in base models before any RL, DeepSeek-V3-Base among them, and traced part of the growth in response length to GRPO's per-answer length averaging (Liu et al., 2025). The Pass@K result is for one 7B model trained on questions from its own supervised data. DeepSeekMath's GRPO uses a learned reward model, while R1's uses rules, so the two papers support GRPO in different regimes.