Reward model overoptimization

Context. RLHF optimizes a learned reward model, an imperfect proxy for human judgment, and pushing it too hard makes outputs worse. The effect was often seen and never measured, since each measurement needs fresh human labels, so the paper lets a large reward model stand in for the humans: a 6B "gold" model labels the comparisons, smaller "proxy" models are trained on those labels, the policy is optimized against a proxy, and the gold score is tracked throughout. Measured in $d=\sqrt{\mathrm{KL}(\pi\|\pi_{\rm init})}$ , gold reward first rises linearly and then turns down as the proxy's errors dominate, and a larger reward model mainly moves the peak further out, smoothly in log parameters. In practice this sets a KL budget that can be predicted from small runs.

Setting. The environment is InstructGPT's: instruction prompts, GPT-3 policies fine-tuned on demonstrations (1.2B unless stated), and GPT-3 reward models with a scalar head. The gold model is InstructGPT's 6B reward model; proxies from 3M to 3B parameters are trained on up to 90,000 comparisons of policy samples, each labeled deterministically by which sample gold scores higher. Gold is centered on the initial policy and has unit variance. Two optimizers are used: best-of-n (BoN), which samples $n$ responses and keeps the one the proxy scores highest, and PPO, with no KL penalty except in one sweep.

Main findings. Gold score follows $R_{\rm bon}(d)=d\,(\alpha_{\rm bon}-\beta_{\rm bon}\,d),\qquad R_{\rm RL}(d)=d\,(\alpha_{\rm RL}-\beta_{\rm RL}\log d),$ with $R(0)=0$ . The coefficients move close to linearly in the log of proxy size, and $\alpha_{\rm RL}$ can be held at one value for all sizes. All of it is empirical, in one environment with one gold model, and the forms are fitted curves with no derivation.

Step 1

The KL of best-of-n has a closed form, and $\sqrt{\mathrm{KL}}$ is the natural axis. If the proxy score has a continuous distribution $F$ under the initial policy, BoN reweights each response by $nF(r)^{n-1}$ , so $\mathrm{KL}_{\rm bon}=\int_0^1 nu^{n-1}\log\big(nu^{n-1}\big)\,du=\log n-\frac{n-1}{n},$ 5.9 nats at $n=1000$ and 10.0 at $n=60{,}000$ . The square root makes $d$ close to linear in RL training time, and it makes linear growth the natural baseline (a gloss, not in the paper): among policies at a given KL the exponential tilt maximizes the expected score, and a Gaussian score with standard deviation $\sigma$ gains exactly $\sigma\sqrt{2\,\mathrm{KL}}$ under it; BoN reaches 94% of that at $n=1000$ .

Step 2

Best-of-n gold is a downward parabola in $d$ , and the form predicted unseen data. Gold rises, peaks and falls while the proxy keeps rising. The form was chosen on data up to $n=1000$ and then tested on a new run to $n=60{,}000$ , a genuine advance prediction. Its peak is at $d^*=\alpha/(2\beta)$ with height $\alpha^2/(4\beta)$ . Read off the coefficient plot ( $\alpha_{\rm bon}$ about 0.50 to 0.67, $\beta_{\rm bon}$ about 0.12 to 0.09 from 3M to 3B), the peak sits near 4 nats with height 0.5 for the smallest proxy and near 14 nats with height 1.2 for the largest, past the measured range.

Step 3

RL gold follows $d(\alpha-\beta\log d)$ , with all of the proxy-size dependence in $\beta$ . PPO runs reach KL near 100 nats and turn down the same way. The log form extrapolated better than forms with finite slope at the origin, and its infinite slope at 0 means it is not meant for small $d$ . Holding $\alpha_{\rm RL}$ fixed leaves a clean trend in $\beta_{\rm RL}$ , from about 0.17 at 3M to about 0.12 at 3B. The peak is at $d^*=e^{\alpha/\beta-1},\qquad R(d^*)=\beta\,d^*,$ so the peak location is exponential in $1/\beta$ , and a modest drop in $\beta$ from a larger proxy gives a much longer stretch of useful optimization. The RL extrapolation is only a held-out check on the same runs; unlike the BoN fit, it was not an advance prediction.

For RL,

\alpha_{\rm RL}

is held fixed across proxy sizes. Why does a modest fall in

\beta_{\rm RL}

matter so much?

Step 4

Reward-model data helps only past a threshold, set by the number of distinct comparisons. Below about 2,000 comparisons every proxy stays near chance validation loss and gains nothing under optimization, whatever its size. Four epochs on 2,000 comparisons match one epoch on 2,000 (validation loss 0.684 against 0.686), while one epoch on 8,000 is clearly better (0.655), so the threshold depends on the number of distinct examples, and more gradient steps on the same examples do not move it. Proxies with equal validation loss give similar BoN gold whatever their mix of size and data, weak evidence (the paper's words) that validation loss alone sets robustness.

Step 5

A larger policy gains less from the same proxy but is exploited about as much. A 6B policy starts with higher gold than a 1.2B one and gains less, yet both peak at nearly the same KL with nearly the same proxy-gold gap; the paper's tentative reading, through Korbak et al. (2022), is that a bigger policy only improves the prior of the tilt.

Step 6

KL does not compare optimizers well, and a KL penalty acts like early stopping. RL needs far more KL than BoN to reach the same gold; by the tilt bound of Step 1, BoN is close to the largest proxy gain possible per nat of KL when the score is roughly Gaussian, so RL's extra KL goes into changes the proxy does not reward. With a 1.2B proxy and penalties from 0.01 to 0.5, gold as a function of KL traces the same curve for every penalty: the penalty raises proxy at a given KL and makes the run converge sooner, so its only effect on gold is where along the curve training stops. The paper flags this as possibly sensitive to PPO tuning.

Step 7

Independent Gaussian noise only shrinks the gain, so a peak requires another mechanism. The paper calls the slope shortfall regressional Goodhart and the peak extremal Goodhart (the proxy failing off its training distribution), a reading its experiments do not test. If the proxy is gold $X$ plus independent noise $Z$ , both Gaussian (or the noise small and bounded, up to an $o(\mathrm{Var}\,Z)$ error), $\mathbb E[X\mid X+Z=c]=\mathbb EX+\frac{\mathrm{Var}\,X}{\mathrm{Var}\,X+\mathrm{Var}\,Z}\,(c-\mathbb EX-\mathbb EZ),$ which rises at a shrunk slope and never falls. For BoN the argument can be made sharper (a gloss, not in the paper): BoN reweights the initial policy by a function of the proxy alone, so it keeps the conditional law of gold given proxy and only shifts the proxy's law upward as $n$ grows, and a falling BoN curve therefore means the initial policy's own highest-proxy answers are worse on gold. Heavy-tailed independent noise does this by itself: with Gaussian gold and Student-t noise with 3 degrees of freedom, $\mathbb E[X\mid X+Z=c]$ peaks near $c=3.3$ and decays back toward 0 (my check), the mechanism behind later results that KL regularization does not contain heavy-tailed reward error (Kwa et al., 2024).

Consequences and limits. Against a fixed proxy there is a KL at which gold peaks, predictable from small runs and moving out only roughly with log reward-model size, so early stopping or a KL target matters in practice and a KL penalty is one way of choosing the stopping point. The paper's sketch of iterated RLHF uses the same accounting: if $\alpha$ and $\beta$ stay fixed and $d$ adds across rounds, $k$ rounds with fresh reward models covering total distance $d$ give $d(\alpha-\beta\log d+\beta\log k)$ , a gain of $\beta d\log k$ that leaves the $\alpha$ part untouched and cannot hold for large $k$ . The main limit is that the gold model is itself a reward model, so the study measures the gap between two reward models and leaves the gap between labels and human intent unmeasured; there is also one environment, two policy sizes and mostly default PPO settings.

Terms

$d=\sqrt{\mathrm{KL}(\pi\|\pi_{\rm init})}$: distance from the initial policy; the axis for every fit
Gold and proxy RM: 6B InstructGPT reward model as ground truth; 3M to 3B models trained on its labels
$\mathrm{KL}_{\rm bon}=\log n-\frac{n-1}{n}$: exact for an atomless proxy score, an upper bound otherwise
$R_{\rm bon}=d(\alpha-\beta d)$: peak at $d^*=\alpha/2\beta$ , height $\alpha^2/4\beta$
$R_{\rm RL}=d(\alpha-\beta\log d)$: peak at $d^*=e^{\alpha/\beta-1}$ , height $\beta d^*$
$\alpha$: linear gold gain per unit $d$ ; read as the part left after regressional shrinkage
$\beta$: rate of overoptimization; falls about linearly in log proxy size
Regressional Goodhart: proxy = gold + independent noise; Gaussian case shrinks gains but stays monotone
Extremal Goodhart: proxy fails off its training distribution; the paper's account of $\beta$
KL penalty: same gold-versus-KL curve; acts as early stopping
About 2,000 comparisons: below this, proxies stay near chance whatever their size
Iterated RLHF: $k$ fresh-RM rounds add $\beta d\log k$ under fixed coefficients and additive $d$

Quiz

In $R_{\rm bon}=d(\alpha-\beta d)$ , what is $\alpha/(2\beta)$ ?
Why can regressional Goodhart with Gaussian gold and Gaussian noise not produce a gold peak?
BoN gold falls at large $n$ . Since BoN reweights $\pi_{\rm init}$ by a function of the proxy alone, what must be true?
Under the paper's assumptions (fixed coefficients, $d$ additive across rounds), what do $k$ rounds of fresh reward models over total distance $d$ change?

Sources

Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. ICML, 2023.

Ahmad Beirami et al. Theoretical guarantees on the best-of-n alignment policy. ICML, 2025.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. RL with KL penalties is better viewed as Bayesian inference. Findings of EMNLP, 2022.
Thomas Kwa, Drake Thomas, and Adrià Garriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. NeurIPS, 2024.