Group Sequence Policy Optimization

Context. In RL with verifiable rewardsreinforcement learning in which a program checks each answer (exact match, unit tests) and returns the reward, a GRPOgroup relative policy optimization: PPO without a value model, in which every token of an answer gets the answer's reward minus its group's mean reward, divided by the group's standard deviation, as advantage step samples a group of answers to each question, scores them, and then takes several gradient steps on that batch, one per minibatch. From the second minibatch on, the model being trained has moved away from the one that sampled the data. GRPO corrects for this as PPOproximal policy optimization: a policy-gradient method whose objective multiplies each token's advantage by the token's probability ratio, current over sampling policy, and clips that ratio so each update moves the policy only a little does, token by token: each token's gradient is weighted by that token's probability ratio, and the token is clipped when its ratio leaves a narrow band around 1. Zheng et al. (2025) report that at large scale this training can collapse beyond repair (rolling back to an earlier checkpoint and retuning did not rescue it), that the problem grows with response length, and that for mixture-of-expertsa transformer in which a learned router sends each token through a few of many feed-forward blocks (the experts) at each layer, so only part of the parameters is active per token models GRPO converged only with an extra fix, Routing Replayrecording which experts the sampling policy used for each token of a sampled answer and forcing the trained policy to use the same experts when the ratio is computed. Their GSPO, group sequence policy optimization, computes one probability ratio per response and clips whole responses.

Zheng et al.'s principle is that the off-policy correction should act on the same unit as the reward. The reward is given to a whole response, so the ratio that corrects for stale samples, and that decides what is clipped, should also belong to the whole response. A long response has thousands of token ratios; each is computed from a single sampled token, each multiplies one token's gradient, and in a mixture-of-experts model each can jump when an update changes which experts the token is routed to. GSPO averages the token log-ratios over the response and exponentiates the average. This smooths out the token-level noise, gives every token of a response the same weight, and keeps or clips the response as a unit. On a mixture-of-experts model with 30B parameters, GSPO trained stably without Routing Replay and reached higher training reward and benchmark scores than GRPO for the same compute, while clipping about 15% of tokens against 0.13% for GRPO.

Setting. For a question $q$, sample $G$the number of answers sampled per question answers $o_1,\dots,o_G$ from $\pi_{\rm old}$the policy as it was when the batch was sampled; $\pi_\theta$, with parameters $\theta$, is the policy being trained, write $o_{i,t}$ for token $t$ of $o_i$ and $|o_i|$ for its length in tokens, and score each answer with a reward $R_i\in[0,1]$. Every token of $o_i$ gets the advantage $\hat A_i$$\hat A_i=(R_i-\bar R)/\operatorname{std}(R)$, with $\bar R$ and $\operatorname{std}(R)$ the mean and standard deviation of the rewards in the group of $q$. Each batch of answers is used for $\mu$the number of optimizer updates made on one batch of sampled answers, one per minibatch; $\mu=1$ is fully on-policy updates ($\mu=4$ in the experiments), so within a batch $\pi_\theta$ drifts away from $\pi_{\rm old}$. Write $\pi_\theta(o_{i,t}\mid\cdot)$ for the probability of $o_{i,t}$ given $q$ and the earlier tokens $o_{i,\lt t}$, and $\rho_{i,t}$${}=\pi_\theta(o_{i,t}\mid\cdot)/\pi_{\rm old}(o_{i,t}\mid\cdot)$, the ratio of token $t$. GRPO maximizes PPO's clipped objective token by token, with clip range $\varepsilon$the clip range: for a token with positive advantage the term stops contributing gradient once $\rho_{i,t}\gt1+\varepsilon$, and for negative advantage once $\rho_{i,t}\lt1-\varepsilon$ and without the KL term, as in the paper:$$\begin{aligned}&J_{\rm GRPO}(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^G\frac1{|o_i|}\sum_{t=1}^{|o_i|}\\&\quad\min\big(\rho_{i,t}\hat A_i,\\&\quad\operatorname{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\hat A_i\big)\Big].\end{aligned}$$GSPO replaces $\rho_{i,t}$ by one ratio per answer, the geometric mean of its token ratios,$$\begin{aligned}s_i(\theta)&=\Big(\frac{\pi_\theta(o_i\mid q)}{\pi_{\rm old}(o_i\mid q)}\Big)^{1/|o_i|}\\&=\exp\Big(\frac1{|o_i|}\sum_{t=1}^{|o_i|}\log\rho_{i,t}\Big),\end{aligned}$$and maximizes$$\begin{aligned}&J_{\rm GSPO}(\theta)=\mathbb E\Big[\frac1G\sum_{i=1}^G\min\big(s_i\hat A_i,\\&\quad\operatorname{clip}(s_i,1-\varepsilon_{\rm low},1+\varepsilon_{\rm high})\hat A_i\big)\Big],\end{aligned}$$with lower and upper clip ranges $\varepsilon_{\rm low}=3\times10^{-4}$ and $\varepsilon_{\rm high}=4\times10^{-4}$GSPO's clip ranges in the experiments; the GRPO baseline uses 0.2 and 0.27 on its token ratios, several hundred times smaller than GRPO's because $s_i$ is a per-token average.

Main findings (empirical apart from the gradient comparison). (a) Without clipping, the two gradients differ only in how the tokens of an answer are weighted: GSPO gives every token of $o_i$ the weight $s_i$, GRPO gives each token its own $\rho_{i,t}$, and on the first minibatch, where both ratios are 1, the two gradients are equal. (b) On a mixture-of-experts model fine-tuned from Qwen3-30B-A3B-Base, GSPO trains stably and keeps improving as compute, fresh questions and longer generation limits are added, with higher training reward and higher scores on AIME 2024the 30 problems of the 2024 American Invitational Mathematics Examination, each with an integer answer; here pass@1 averaged over 32 samples per problem, LiveCodeBencha benchmark of recent competitive-programming problems scored by hidden tests; here problems from October 2024 to February 2025, pass@1 averaged over 8 samples and CodeForcesa competitive-programming site; the score is an Elo rating estimated from the model's solutions to its contest problems than GRPO at the same compute (curves only, no tables). (c) Averaged over training, GSPO clips 15% of tokens and GRPO 0.13%, so GSPO learns more from fewer tokens. (d) In this model about 10% of the experts used for a given answer change after one gradient update; GRPO fails to converge unless the routing is frozen with Routing Replay, while GSPO converges without it.

Consequences. For RL on long responses, and in particular on mixture-of-experts models, the change to a GRPO implementation is small: average each answer's token log-ratios, exponentiate, clip that number with ranges of order $10^{-4}$, and weight all the answer's tokens by it. A clip fraction is not comparable between token and sequence clipping, so a high fraction under GSPO is not by itself a sign of trouble. The authors also expect GSPO to tolerate the small numerical differences between a training engine and an inference engine, since only an answer's average log-ratio enters, which would let one take $\pi_{\rm old}$'s log-probabilities from the inference engine instead of recomputing them with the trainer; this is argued without a measurement. Where tokens of one answer should get different advantages, as in multi-turn RL, a token-level variant keeps the sequence ratio (Step 5).

Limits. The evidence is one model (a 30B mixture-of-experts model after a cold start), one GRPO baseline and training curves, with no tables, seeds or error bars. Because GRPO needs Routing Replay in this setup, the main comparison is GSPO without it against GRPO with it; how much of GSPO's gain remains on a dense model, where there is no routing to change, is not shown. The claim that GSPO can use the inference engine's likelihoods directly is not tested, and the reason given for GRPO's collapse (token-level noise that accumulates with length) is an interpretation of the curves rather than a measurement of gradient variance.