Entropy collapse in RL for reasoning

Policy at one state
$s;$ $a$	state (prompt plus tokens so far); next token
$\pi_a,$ $z_a$	probability and logit of token $a$ at $s$
$A_a$	advantage of token $a$ ; averages to zero under $\pi$
$H(s)$	entropy $-\sum_a\pi_a\log\pi_a$ at $s$
$H_t,$ $H^i_t$	token entropy at position $t$ ; the same in response $i$
$\bar H$	policy entropy: $H_t$ averaged over a batch
Updates
$\Delta z_a;$ $\eta$	logit change in one update; step size
$\pi_{\rm old}$	policy that sampled the responses
$\epsilon_{\rm low},$ $\epsilon_{\rm high}$	lower and upper clipping widths of the probability ratio
$\mathrm{Cov}_i$	centered product of log-probability and advantage for token $y_i$ , the selection score of Step 6
$\ell^i_t$	DAPO's clipped term for token $t$ of response $i$
Constants
$R;$ $R_0,$ $\kappa$	validation accuracy; coefficients of Cui et al.'s fit $R=R_0-\kappa e^{\bar H}$ , fitted per run
$\alpha$	entropy-bonus weight in Step 5
$\beta$	KL-Cov penalty weight, set to 1
$\rho;$ $\tau_\rho$	fraction of tokens kept in Wang et al.'s objective (20%); its per-batch entropy threshold

Context. In RLVRreinforcement learning with verifiable rewards: RL on problems whose answers a program checks, with reward 1 for a correct final answer and 0 otherwise the model's policy entropythe average, over the tokens of sampled responses, of the entropy of the model's next-token distribution; near 0 means the model is almost sure of every token drops sharply in the first steps and keeps falling to near zero unless something stops it, and validation accuracy stalls at about the same time. A nearly certain policy samples nearly the same response every time, so it stops exploring, and with advantageshow much better a response's reward is than the average; here the average is over the group of responses sampled for the same prompt computed within a group of samples, identical samples give no learning signal. Cui et al. (2025) measure how accuracy trades against entropy, derive why the entropy falls, and keep it up by restraining the updates of a handful of tokens. Wang et al. (2025) look at which tokens carry the entropy and find that a policy-gradient update restricted to the 20% of tokens with the highest entropy matches or beats the full update.

One policy-gradient step lowers the entropy at a position by about the covariance, over the next-token distribution, between a token's log-probability and the change in its logit (the network output before the softmax). Policy gradient raises each logit in proportion to the token's advantage (and to its probability, for the standard policy gradient). Starting from a pretrained model, the tokens the model already finds likely are usually the ones that lead to reward, so the covariance is positive and the entropy falls step after step; it rises only when a rare token earns reward or a likely one is penalized. The covariance vanishes wherever the next token is nearly certain, so the whole effect plays out at the few positions where the model is genuinely uncertain. In a long chain of thought these are few: half the tokens have entropy below 0.01, and the most uncertain fifth are words like "wait", "however" and "suppose", where the reasoning can branch. In these experiments, controls that act on those few tokens work, and controls applied to every token (an entropy bonus, a KL penalty to the starting model) do not.

Setting. Fix a state $s$ the prompt together with the tokens generated so far; the next token is the action and write $\pi_a$ the probability the policy gives next token $a$ at the state $s$ fixed in the text for the probability of next token $a$ , $z_a$ the logit of token $a$ at that state, the network output before the softmax, so $\pi_a=e^{z_a}/\sum_b e^{z_b}$ for its logit and $A_a$ the advantage of token $a$ : how much better than average the outcome is when $a$ is chosen; it averages to zero under $\pi$ for its advantage. The entropy at $s$ is $H(s)=-\sum_a\pi_a\log\pi_a$ . Wang et al. call $H_t$ the token entropy at position $t$ of a response, $H(s_t)$ , the entropy of the distribution the $t$ -th token is drawn from; $H^i_t$ is the same for response $i$ the token entropy at position $t$ , and Cui et al.'s policy entropy $\bar H$ the token entropy $H_t$ averaged over the tokens of responses sampled for a batch of training prompts is its average over a batch. Cui et al. run zero-RLRL applied directly to a pretrained base model, with no supervised fine-tuning first on 11 models from four families (Qwen2.5 from 0.5B to 32B, Mistral, LLaMA and DeepSeek-Math) on math and code, with GRPOgroup relative policy optimization: a PPO-style method with no value function, whose advantage is the response's reward standardized within the group of responses to the same prompt, given to every token of the response and three other algorithms, 8 responses per prompt, no KL penalty or entropy bonus, and prompts whose responses are all right or all wrong filtered out. Wang et al. train Qwen3 base models at 8B, 14B and 32B on 17,000 math problems with DAPOa GRPO variant with clip-higher, dynamic sampling (dropping prompts whose responses are all right or all wrong), a loss averaged over tokens and a penalty for overlong responses, using clip-higherPPO's clipping range $[1-\epsilon_{\rm low},1+\epsilon_{\rm high}]$ on the probability ratio (a token's probability under the policy being updated over its probability under the policy that sampled it), with $\epsilon_{\rm high}\gt\epsilon_{\rm low}$ , so tokens with positive advantage can grow more per update with $\epsilon_{\rm low}=0.2$ and $\epsilon_{\rm high}=0.28$ , no KL or entropy terms and a 20,480-token limit, and report accuracy averaged over 16 samples at temperature 1 on six math benchmarks.

Main results. Without intervention, validation accuracy is a fixed function of entropy over the whole run, so the final accuracy can be predicted from the first 15% of training and the ceiling is the accuracy at zero entropy (Step 1). To first order the entropy change at a state is minus a covariance of log-probability and logit change (Step 2), policy gradient makes the logit change proportional to the advantage, times the probability for the plain gradient (Step 3), and the measured covariance tracks the measured entropy drop and stays positive (Step 4). An entropy bonus is unstable and a KL penalty costs accuracy (Step 5), while restraining the updates of the 0.02% to 0.2% of tokens with the largest covariance keeps entropy more than ten times higher and raises the 32B model's average on seven math benchmarks from 45.8 to 52.2 (Step 6). Wang et al. find that the high-entropy tokens are the branch points of the reasoning, that after RLVR they are mostly the same tokens as in the base model, and that RLVR changes entropy mostly there (Steps 7 and 8), and that a gradient on only the top 20% by entropy gains 7.7 and 11.0 points on AIME24 and AIME25 at 32B and matches the full update at 8B, while the bottom 80% alone does much worse (Step 9).

Consequences and limits. Without entropy control, RL on a pretrained model moves along a fixed curve from entropy to accuracy, and most of the movement is early: in Cui et al.'s runs, 73% of the entropy drop and 76% of the accuracy gain come in the first 200 of 2,400 gradient steps. Running such a recipe longer adds little, so long runs need entropy control, and the control has to be targeted, because the entropy that matters sits on a small set of tokens. Clip-higher does this indirectly: the upper clip binds only on tokens with positive advantage, and raising it lets rare tokens with positive advantage grow; such updates have negative covariance and so raise entropy (their average covariance is about $-0.03$ by Cui et al.'s measurement). Clip-Cov, KL-Cov and Wang et al.'s top-20% mask do it directly. Entropy collapse is one cause of the narrowing of pass@ $k$ the chance that at least one of $k$ independent samples is correct, averaged over problems that Yue et al. (2025) report after RLVR, though there raising the temperature to restore the entropy closes only part of the gap. The theory is first order, and Step 3 is tabular: it treats each state's logits as free parameters, while in a network one update moves the logits at all positions together, so Step 3 gives the direction of the effect, and the measurements of Step 4 test it in a network. Cui et al. note that other models and off-policy data show different entropy curves, so the fitted law is not universal, and neither paper finds an optimal entropy level. Wang et al.'s main results are on Qwen3 trained on math only (LLaMA-3.1-8B needed a supervised warm start, after which the top-20% run also led, at low accuracy), and the best fraction, 20%, may differ elsewhere.

Step 1 (Cui et al.)

Entropy collapses early, and accuracy follows it along one curve. In every run the policy entropy drops sharply at the very start and keeps falling toward zero, while validation accuracy rises fast and then saturates; averaged over the 11 models, 73% of the entropy drop and 76% of the accuracy gain come in the first 200 of 2,400 gradient steps, and 94% and 93% by step 800. Plotted against entropy, measured every 4 rollout steps, validation accuracy $R$ follows $R=R_0-\kappa\,e^{\bar H}$ in every run, math and code alike, with $R_0,\ \kappa$ the two coefficients fitted per run (the paper's $b$ and $a$ ): $\kappa$ is the rate $-dR/d\bar H$ at zero entropy, and $R_0-\kappa$ is the accuracy at zero entropy fitted to over 200 points. Fitting only the first 36 steps of the Qwen2.5 runs predicts the next 200 with a root-mean-square error of 0.9 accuracy points on math and 1.2 on code. The fitted curves hardly change between GRPO, RLOO, PRIME and REINFORCE++, and for Qwen2.5 both coefficients vary log-linearly with model size, so a larger model's curve can be extrapolated from smaller ones. The ceiling $R_0-\kappa$ at $\bar H=0$ is fixed by the model and the data, which is the authors' reading of the law: without entropy control, more RL compute cannot go past it.

Step 2 (Cui et al.)

To first order, an update changes the entropy at a state by minus a covariance. For a softmax, $\partial\log\pi_a/\partial z_b=\mathbb 1[a=b]-\pi_b$ , so $\partial H/\partial z_a=-\htmlData{gr=g5}{\pi_a}(\log\pi_a+H)$ , and since $\mathbb E_{a\sim\pi}[\log\pi_a+H]=0$ , the sum below is a covariance; a smallwhy: the identity is a first-order Taylor expansion of the entropy in the logits; a large step adds second-order terms, which the approximation ignores logit change $\Delta z_a$ the change in the logit $z_a$ made by one update of the policy changes the entropy by $\begin{aligned}\Delta H&\approx-\sum_a\pi_a(\log\pi_a+H)\,\Delta z_a\\&=-\mathrm{Cov}_{a\sim\pi}\big(\log\pi_a,\ \Delta z_a\big).\end{aligned}$ The identity holds for any small logit change, whatever produced it. Raising the logit of a token that is already likely concentrates the distribution and lowers the entropy; raising the logit of an unlikely one spreads it and raises the entropy. Cauchy-Schwarz bounds the covariance by the standard deviation of $\log\pi_a$ under $\pi$ times that of $\Delta z_a$ , and the first goes to zero as $\pi$ approaches a point mass, so a state whose next token is nearly certain changes its entropy very little under any small update (my remark, not in either paper; it is the link to Wang et al.'s Step 8).

Step 3 (Cui et al.)

Policy gradient moves each logit in proportion to the advantage, so entropy falls when the likely tokens are the rewarded ones. With a separate logit for every state and tokenwhy: the exact gradient formulas need each logit to be its own parameter (a tabular policy); in a network, parameters are shared across positions, so they hold only approximately, which Step 4 checks, the gradient of the expected advantage $\sum_b\pi_bA_b$ with respect to $z_a$ is $\pi_a(\htmlData{gr=g7}{A_a}-\mathbb E_\pi A)=\pi_aA_a$ , since advantages average to zero under the policy, so a gradient step of size $\eta$ the step size (learning rate) of the update gives $\Delta z_a=\eta\pi_aA_a$ . A natural policy gradient step, which preconditions by the Fisher information, gives $\Delta z_a=\eta A_a$ instead (a standard result for softmax policies; Agarwal et al., 2021). Substituting into Step 2, $\begin{aligned}\Delta H_{\rm PG}&\approx-\eta\,\mathrm{Cov}_{a\sim\pi}(\log\pi_a,\pi_aA_a),\\\Delta H_{\rm NPG}&\approx-\eta\,\mathrm{Cov}_{a\sim\pi}(\log\pi_a,A_a).\end{aligned}$ Entropy falls when high-probability tokens have high advantage or low-probability tokens low advantage, and rises in the opposite cases. On problems it can partly solve, a pretrained model usually gives its highest probabilities to the tokens that lead to reward, so the covariance is positive on average and the entropy falls at every step. Cui et al. treat PPO's clipped updates as closer to the natural-gradient form, since both limit the step in KL, and use that form in Step 6.

Step 4 (Cui et al.)

The measured covariance tracks the entropy drop and stays positive. To test Step 3 they train Qwen2.5-7B with on-policy GRPO without PPO's surrogate loss, treating each prompt as a state and each whole response as an action, with the response's log-probability divided by its length; the covariance of log-probability and $\pi_aA_a$ is computed within each prompt's group of responses and averaged over the batch. The per-step entropy decrease and the covariance follow nearly the same curve, both large at the start and settling at a lower level, and the covariance stays positive throughout, which is why the entropy never recovers. Grouping prompts by accuracy (the fraction of their 8 responses that are correct), the covariance is largest for easy prompts (accuracy 0.875) and smallest for hard ones (0.125); on hard prompts, what the model finds likely is not reliably what earns reward.

Step 5 (Cui et al.)

An entropy bonus and a KL penalty are poor controls. Adding $\alpha\bar H$ to the objective, with $\alpha$ the weight of the entropy bonus set to 0.0001 or 0.001, barely changes the entropy; at 0.01 the entropy explodes, and at 0.005 it holds steady without beating the baseline's accuracy. A KL penalty to the reference model, with weights from 0.001 to 0.1, also keeps the entropy stable but lowers accuracy. Both act on every token alike, while the entropy change is concentrated on a few (Steps 6 and 8).

Step 6 (Cui et al.)

Restraining the few tokens with the largest covariance keeps entropy up and raises accuracy. For each of the

N

response tokens

y_i

of a batch, with advantage

A_i

and with bars denoting batch means, they compute the centered product of log-probability and advantage,

\mathrm{Cov}_i=\big(\log\pi(y_i)-\overline{\log\pi}\big)\big(A_i-\bar A\big),

a per-token proxy for the natural-gradient covariance of Step 3. It is extremely skewed: at the first step of Qwen2.5-7B it averages 0.003 over all tokens, 1.385 over the top 2% and 5.654 over the top 0.02%. Clip-Cov removes from the gradient a random

2\times10^{-4}

of the tokens, drawn from those with

\mathrm{Cov}_i

between 1 and 5, both far above the mean. KL-Cov instead adds a penalty

\beta\,|\log\pi-\log\pi_{\rm old}|

, with

\pi_{\rm old}

the policy that sampled the responses, before this round of updates the sampling policy and

\beta=1

, on the top

2\times10^{-3}

of tokens by

\mathrm{Cov}_i

at 7B (

2\times10^{-4}

at 32B); in plain PyTorch,

# logp, old_logp, adv: one entry per response token in the batch
ratio = torch.exp(logp - old_logp)
loss = -ratio * adv                       # policy-gradient term per token
cov = (logp - logp.mean()) * (adv - adv.mean())
n_sel = int(0.002 * logp.numel())         # top 0.2% of tokens (7B)
idx = torch.topk(cov.detach(), n_sel).indices
mask = torch.zeros_like(loss)
mask[idx] = 1.0
kl = (logp - old_logp).abs()
loss = loss + beta * mask * kl            # beta = 1
loss = loss.mean()

Trained with GRPO on DAPO's math data, KL-Cov keeps the entropy more than ten times above the baseline's plateau, lengthens responses, and raises the seven-benchmark average from 38.6 to 40.6 at 7B and from 45.8 to 52.2 at 32B (AIME24 from 21.8 to 36.8, AIME25 from 16.2 to 30.8). Clip-Cov gives 40.4 and 50.3, and clip-higher 38.8 and 47.2, with an entropy that rises early and then becomes unstable. The entropy level can be set by the clipped fraction or the KL weight, but the authors find no relation between the level reached and the final accuracy.

Step 7 (Wang et al.)

Most tokens of a chain of thought are nearly certain, and the uncertain few are branch points. Over more than a million tokens of Qwen3-8B reasoning on AIME problems at temperature 1, 50.6% of tokens have entropy below 0.01, and only the top 20% exceed 0.672. The frequent tokens with the highest average entropy connect or redirect the reasoning ("wait", "however", "thus", "since", "suppose", "assume"), and those with the lowest complete a word, a formula or a line of code. Decoding with one temperature for tokens above the 0.672 threshold and another for the rest shows the roles: lowering it on the high-entropy tokens hurts AIME accuracy much more than lowering it on the others, and raising it on them helps, while raising it on the others degrades the output and eventually produces nonsense. Branching happens at the high-entropy tokens, and keeping them uncertain helps; Wang et al. call them forking tokensthe high-entropy tokens of a chain of thought, here the top 20% by entropy, at which the reasoning can go in different directions.

Step 8 (Wang et al.)

RLVR keeps the base model's forks and changes entropy mostly there. Through DAPO training of Qwen3-14B, the positions in the top 20% by entropy still overlap with the base model's top 20% at 86.7% at the end of training (step 1360). Grouping tokens by the base model's entropy in 5% bands, the change in entropy after RL grows steadily with the starting entropy, and the lower percentiles of the entropy distribution hardly move during training. This is what the bound in Step 2 predicts: to first order, the entropy of a nearly certain token cannot move much.

Why do tokens whose next-token distribution is nearly one-hot barely change entropy during RL, whatever their advantage?

Step 9 (Wang et al.)

A gradient on only the top 20% of tokens by entropy matches or beats the full gradient. Write $\ell^i_t$ DAPO's clipped per-token term for token $t$ of response $i$ : $\min\big(r^i_t\hat A^i,\ \mathrm{clip}(r^i_t,1-\epsilon_{\rm low},1+\epsilon_{\rm high})\hat A^i\big)$ , with $r^i_t$ the probability ratio to the sampling policy and $\hat A^i$ the response's group-standardized reward for DAPO's per-token term and $\tau_\rho$ the entropy threshold above which a fraction $\rho$ of the batch's tokens lie, recomputed for every batch for the threshold of the top fraction $\rho$ . The objective keeps only those tokens and averages over them, $J_\rho=\frac{\sum_{i,t}\mathbb 1[H^i_t\ge\tau_\rho]\,\ell^i_t}{\sum_{i,t}\mathbb 1[H^i_t\ge\tau_\rho]}.$ At $\rho=20\%$ the 32B model reaches 63.5 on AIME24 and 56.7 on AIME25, against 55.8 and 45.6 with all tokens; at 14B the gains are 5.2 and 4.8 points; at 8B the two are level (six-benchmark averages of 54.2 and 53.7). Training on the bottom 80% alone does much worse, although it uses four times as many tokens. At 14B and 32B, $\rho=10\%$ is slightly worse than 20%, and 100% clearly worse, and the overall entropy during training is highest at 20%, which the authors read as the fraction that best supports exploration. The masked runs produce longer responses and do better on LiveCodeBench although trained only on math, and raising the 32B model's limit to 29,696 tokens lifts AIME24 to 68.1. The same view explains why an entropy bonus underperforms: it raises the entropy of the low-entropy majority, which Step 7 shows hurts, while clip-higher mostly reaches high-entropy tokens, since tokens with large probability ratios tend to have high entropy.

Technical notes Cui et al.'s summary box writes the law as

R=-a\exp(H+b)

, while their equation and abstract give

R=-ae^{H}+b

, the form fitted (here

R_0=b

\kappa=a

). Their statement before Proposition 1 writes the update as

z-\eta\nabla J

, a descent step, while the proof uses ascent, and the proposition omits the state-visitation weight of the tabular policy gradient, which only rescales the step size at each state. They call the Clip-Cov bounds of 1 and 5 over 500 times the average covariance, while their own average of 0.003 puts the lower bound near 330 times. Their Eq. 10 centers with batch means, so its expectation is the covariance pooled over all positions, which adds a between-position term to the per-state covariance of Step 3 (law of total covariance); the selection uses a proxy, not an estimate, of that covariance.

Terms

Entropy collapse: policy entropy falls to near 0 early in RL and accuracy saturates with it
Entropy-accuracy law: $R=R_0-\kappa e^{\bar H}$ per run; ceiling $R_0-\kappa$ at $\bar H=0$ ; predictable from early steps
Covariance identity: $\Delta H\approx-\mathrm{Cov}_{a\sim\pi}(\log\pi_a,\Delta z_a)$ at a state, for any small logit change
Logit updates: $\Delta z_a=\eta\pi_aA_a$ for policy gradient, $\eta A_a$ for natural policy gradient (tabular softmax)
Why entropy falls: the likely tokens are usually the rewarded ones, so the covariance is positive
Near-certain tokens: the covariance is at most the standard deviation of $\log\pi_a$ under $\pi$ times that of $\Delta z_a$ , so their entropy barely moves
Clip-Cov, KL-Cov: detach, or KL-penalize, the 0.02% to 0.2% of tokens with the largest $\mathrm{Cov}_i$
Clip-higher: raising $\epsilon_{\rm high}$ lets rare tokens with positive advantage grow, which raises entropy
Forking tokens: the top 20% of tokens by entropy (above 0.672 for Qwen3-8B); connectives where reasoning branches
Top-20% gradient: AIME24/25 +7.7/+11.0 at 32B, +5.2/+4.8 at 14B, level at 8B; bottom 80% alone much worse
Entropy bonus: acts on every token, raising entropy where it hurts; unstable or ineffective in Cui et al.'s runs; Wang et al. argue it targets the wrong tokens

Quiz

What does the fitted law $R=R_0-\kappa e^{\bar H}$ imply for spending more compute on an RL run without entropy control?
Wang et al. argue that an entropy bonus is the wrong way to encourage exploration in RLVR. Why?
How does clip-higher raise policy entropy?

Sources

Ganqu Cui, Yuchen Zhang, Jiacheng Chen, et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv preprint, 2025.
Shenzhi Wang, Le Yu, Chang Gao, et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS, 2025.

Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift. Journal of Machine Learning Research, 2021.
Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint, 2025.
Yang Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS, 2025.