Scaling RL compute

Context. In pretraining, loss fitted against compute on small runs predicts the loss of large ones. RL for reasoning, now run at budgets such as the 100,000 GPU-hours reported for DeepSeek-R1-Zero, has had nothing similar. Recipes such as GRPOgroup relative policy optimization: PPO-style clipped updates without a value function, where each sample's advantage is its reward standardized within the group of samples for the same prompt and DAPOa GRPO variant with separate lower and upper clipping thresholds, per-prompt averaging of the token losses, and removal of prompts whose samples all get the same reward differ in many choices (the loss, how advantages are normalized, how stale the samples may be, which prompts are kept), and they are compared at one budget, often on downstream benchmarks, so it is unknown whether a choice that helps at that budget helps at a larger one. Khatri et al. spend more than 400,000 GPU-hours on RL for reasoning, mostly math with an 8B model, fit a curve of validation pass rate against compute for each variant, and check the fits by extending the runs. The curve that works is a sigmoid in log compute with a ceiling, and a recipe choice should be judged first by what it does to that ceiling, since a choice that only speeds up early training can be overtaken later. This is the RL counterpart of the compute-scaling fits in the scaling-laws notes, with a bounded pass rate in place of a loss.

The ceiling and the speed of approach separate cleanly. The loss function and the numerical precision of the output layer move the ceiling. The asynchronous setup, the averaging of the loss, the normalization of advantages and the choice of prompts mostly change the speed and move the ceiling by no more than its run-to-run error. Batch size and generation length raise the ceiling at the cost of a slower start, and a larger model raises it and leads from the start. Combined, the choices form ScaleRL, whose fitted ceiling of 0.61 exceeds those of the GRPO, DAPO and Magistralthe RL recipe published for the Magistral reasoning models: DAPO's loss with streaming asynchronous generation recipes (0.49 to 0.535) and equals that of MiniMaxthe RL recipe published for the MiniMax-M1 model, with a truncated importance-sampling loss and a 32-bit output layer with a faster rise, and a 100,000 GPU-hour run of it follows the curve fitted to its first half.

Setting. The curve relates $R_C$the validation pass rate after compute $C$: mean correctness of 16 samples per prompt on 1,000 prompts held out from the training data to training compute $C$GPU-hours spent on RL so far, for a fixed model and training setwhy: a fitted curve describes one recipe on one model and one prompt set; Step 6 shows that changing the model or the batch moves it. It writes the gain over the starting pass rate $R_0$the pass rate at the start of the fitted range as the gain at the ceiling times the fraction of it reached: $$\begin{aligned}R_C-R_0&=\underbrace{(A-R_0)}_{\text{gain at the ceiling}}\\&\quad\times\underbrace{\frac{1}{1+(C_{\rm mid}/C)^B}}_{\text{fraction reached}}.\end{aligned}$$ The ceiling $A$the pass rate the run approaches as compute grows, between 0 and 1 is what a recipe can reach; $C_{\rm mid}$the compute at which half of the gain $A-R_0$ has been reached and the exponent $B$the steepness of the rise: the fraction reached is the logistic function $1/(1+e^{-z})$ of $z=B(\log C-\log C_{\rm mid})$ set how fast it gets there. Fits start after about 1.5k GPU-hourswhy: in roughly the first epoch the pass rate rises almost linearly before the sigmoid regime, and less stable setups often plateau early there; including it made fits noisier. The prompts are the math problems of Polaris-53Ka public set of about 53,000 math problems with checkable answers, used for RL, the model is an 8B dense model after supervised fine-tuning on reasoning traces, and the reward is $\pm1$ from an automatic answer checker. The base recipe is GRPO without a KL penalty and with DAPO's asymmetric clipping. For a prompt $x$, the generating copy of the model $\pi_{\rm gen}$the copy of the model that samples completions, running fast inference kernels on separate GPUs samples $G$the number of completions per prompt, 16 completions $y_i$ with rewards $r_i$; the advantage is the group-standardized reward $\hat A_i=(r_i-\bar r)/\mathrm{std}(r)$, with $\bar r$ and $\mathrm{std}(r)$ the mean and standard deviation of the group's rewards (the hat distinguishes it from the ceiling $A$); and each token's importance ratio $\rho_{i,t}$the importance ratio $\pi_{\rm train}(y_{i,t})/\pi_{\rm gen}(y_{i,t})$ of token $t$ of completion $i$ between the copy being trained and the copy that sampled it$=\pi_{\rm train}(y_{i,t})/\pi_{\rm gen}(y_{i,t})$ is clipped to $[1-\epsilon^-,1+\epsilon^+]$DAPO's clipping range, with a larger upper threshold than lower one. A batch is 48 prompts (768 completions), generations have 12,288 thinking tokens, and runs use 80 GPUs, 64 generating and 16 training.

Main findings (all empirical). Fitted on the same setup, ScaleRL reaches $A=0.61$ with $B=1.97$, MiniMax's $0.61$ with $B=1.77$, Magistral's 0.535, DAPO's 0.515 and GRPO's 0.49. Three independent ScaleRL runs vary by at most $\pm0.015$ in $A$, so differences under 0.02the error margin on the ceiling $A$, taken from the spread of three independent runs of ScaleRL are treated as noise. In forward ablations (one choice of the base recipe changed at a time, in runs of 3.5 to 4k GPU-hours), the loss (DAPO 0.52, GSPOgroup sequence policy optimization: the same clipped objective with one importance ratio per completion, the ratio of its sequence probabilities, in place of token ratios 0.59, CISPOclipped importance sampling policy optimization: plain policy gradient with each token's importance weight capped at a fixed value and held constant 0.595) and FP3232-bit floating point, in place of the lower precision used elsewhere logits (0.52 to 0.61) move $A$ clearly, while the asynchronous setup changes only $B$ (2.68 to 4.55 at the same 0.52). Leaving any one component out of ScaleRL keeps $A$ between 0.59 and 0.61 and lowers efficiency (the steepness $B$). At larger scale, a batch of 2048 completions against the default 768 raises $A$ to 0.645, generations of 32k tokens raise it to 0.645 with a much later midpoint, and a 17B×16 mixture-of-expertsa model whose layers route each token to a few of many expert subnetworks; 17B×16 means 16 experts and 17B parameters active per token model reaches 0.71 and passes the 8B model's final score with a sixth of its compute. On the 100,000 GPU-hour 8B run, the sigmoid fitted up to 50,000 GPU-hours predicts $A=0.645$ and the extended run follows it.

Consequences and limits. The method lets a practitioner compare recipes before running the large job: train each candidate to a few thousand GPU-hours, fit the curve on held-out pass rate, prefer the higher ceiling and only then the faster rise, and watch the share of generations cut off at the length limit, since runs at batch 768 whose truncation reached 10 to 15% destabilized. The choices that held up were CISPO's loss, FP32 at the output layer, streaming asynchronous generation, dropping prompts whose samples all score the same, retiring prompts the model solves 90% of the time, and large batches. The evidence comes from two models (the 8B dense model and one mixture-of-experts model), mostly one math dataset, and in-distribution validation; downstream scores such as AIMEproblems from the American Invitational Mathematics Examination, a standard test of competition math moved with validation but were not modeled. The forward ablations were mostly fitted on runs of 3.5 to 4k GPU-hours, so their ceilings are long extrapolations, and the 0.02 margin comes from three runs of one recipe.