PPO and RLOO for language models

The KL term keeps outputs where the reward model is valid; without it the policy writes gibberish. In a mock task whose reward is a sentiment classifier's log odds, training with no KL penalty, even with an entropy bonusa reward for keeping the policy's per-token distribution spread out, which discourages it from collapsing onto a few tokens targeting 30 nats, gives text like "These These These sound flowed instantly easily easily" that scores about $+8$ log odds (99.97% positive). The KL term acts as an entropy bonus, keeps the policy where $r_\phi$ was trained, and for style tasks supplies coherence the labelers were never asked about. Because runs with the same $\beta$ ended at very different distances from the start, Ziegler et al. also steered $\beta$ toward a target KL with a log-space proportional controlleran update of $\beta$ by a factor proportional to the relative error between the measured KL and its target, with that error clipped to $\pm20\%$, so $\beta$ changes by at most 2% per step,$$\begin{aligned}u_t&=\frac{\mathrm{KL}_t-\mathrm{KL}_{\rm target}}{\mathrm{KL}_{\rm target}},\\e_t&=\operatorname{clip}(u_t,-0.2,0.2),\\\beta_{t+1}&=\beta_t(1+0.1\,e_t),\end{aligned}$$where $\mathrm{KL}_t$ is the KL from the starting model measured at step $t$ and $u_t$ its relative error. With the reward known, the best policy at each KL is the reference multiplied by $e^{r/\beta}$ and renormalized, the optimum of the KL-regularized objective, and after 2M episodes PPO is short of it, a gap that more training mostly closes. Other details that recur later: the reward model's outputs are normalized to mean 0 and variance 1 on samples from the starting model, and a response missing the required end marker (a period for style tasks, a newline for summaries) gets a fixed reward of $-1$.

The policy learns what labelers check, including their shortcuts. Summarization models trained on 60k labels copy whole sentences from the input 71% of the time on TL;DR and 98% on CNN/Daily Maila news summarization dataset of articles paired with bullet-point highlights, while choosing which sentences to copy and skipping preambles. Labelers prefer them to the human reference summaries (96% and 84%) yet not to lead-3a baseline that copies the first three sentences of the article (45% and 40%). On 30 articles per dataset the authors judge 26 and 29 of these summaries accurate, against at most 20 for the more abstractive supervised models. Copying is an easy way to be accurate, the labelers were told to penalize inaccuracy but not copying, and checking for copying is fast, so the reward model rewarded it. Collecting labels on the current policy's samples during training (online collection) mattered for summarization, where labelers preferred the online model to an offline one 64% (TL;DR) and 65% (CNN/Daily Mail) of the time, and not for the style tasks. A refactor that flipped the sign of both the reward and the KL penalty produced fluent text that scored as badly as possible under the labelers' instructions, since the KL term still kept it grammatical.

The reward is defined only at the end-of-sequence token, so EOS handling decides whether PPO gets a valid signal. The reward model is trained to read its score at the EOSthe end-of-sequence token that ends a response token. If padding uses the EOS token itself, a common shortcut, EOS is masked as padding during supervised training and the model never learns to stop. The reproduction keeps a separate padding token and, following what the original code appears to do, samples a fixed 53 tokens, truncates at the first EOS, and gives $-1$ to any response with no EOS, the "EOS trick", which both defines every reward and favors responses that end. Dropout is turned offwhy: with dropout active, the log-probabilities of the same tokens differ between the sampling pass and the training pass, so the ratio is not 1 at the start of the first epoch and the KL estimate is unreliable in all stages. Prompts are truncated at a paragraph boundary to fit 512 tokens, and are padded on the left for generation, while the reward model was trained on right-padded sequences; scoring the left-padded form changes the 6.9B reward model's scores by about $5\times10^{-4}$ on average.

The reward model and the value model are initialized from trained models, and their outputs are kept on a fixed scale. The reward model is the supervised model with a linear head whose weights are drawn with standard deviation $1/\sqrt{d_{\rm model}+1}$ (the paper writes $N(0,1/\sqrt{d_{\rm model}+1})$ with the standard deviation in the second slot), with $d_{\rm model}$ the hidden width, trained for one epoch, and then shifted so the human reference summaries score 0 on average. The value model is initialized from the reward model, which makes the early value estimates and so the early policy gradients sensible, and it ends up acting like a per-token reward model. Advantages are whitenednormalized within each batch to mean 0 and variance 1 (in the reward version the paper prints, the batch mean is added back) in each batch. Whitening the rewards as well gives shorter summaries and a lower raw win rate, but the same win rate at a given length. The PPO settings are $\beta=0.05$, $\gamma=1$, $\lambda=0.95$, four passes over each batch of 512, $\varepsilon=0.2$, the value loss clipped at 0.2 with weight 0.1, sampling temperature 0.7, and 1M episodes, about 8.6 passes over the 117k training prompts.

The reproduction scales with model size, and its failures are over-optimization and length. Win rates against the reference summaries rise with model size, and the 2.8B and 6.9B models beat the 1.3B checkpoint of the original work. Two 1B runs drifted to a KL of about 50 and 85 nats from the start: their KL-shaped reward was higher, yet the judge preferred them to the references less than 20% of the time, and they wrote run-together strings with no spaces, the over-optimizationa learned reward that keeps rising under optimization while the true quality it stands in for falls measured by Gao et al. PPO summaries are longer than supervised ones, a confound the paper addresses by binning by length relative to the reference; PPO wins in every bin, and its win rate rises with length, which means either that the judge favors length or that longer summaries are better. A DPO model trained on the same preference data, read as an implicit reward model, has lower validation accuracy than the explicit reward model at every size and seed.

Lowering GAE's $\lambda$ makes the advantage less noisy but biased, and the reward falls. With $\gamma=1$ the sum in GAE telescopes at $\lambda=1$ to the observed return minus the value,$$\hat A_t^{\lambda=1}=\sum_{t'\ge t}R_{t'}-V(\htmlData{gr=gn7}{s_t}),$$so the value model is only a baseline and the advantage is unbiased but carries the full variance of the sampled return, while at $\lambda=0$ the advantage $R_t+V(s_{t+1})-V(s_t)$ relies on the value model's prediction of the next prefix, which lowers the variance and adds bias wherever that prediction is wrong. On helpful-harmless data with Llama 7B, PPO's reward curve falls in order as $\lambda$ goes from 1 to 0.95, 0.5 and 0, the opposite of what one would see if variance were the main problem. They attribute this to how concentrated the supervised model's predictions already are: after the first token, the supervised model puts about 60% of its probability on the single most likely next token and over 90% on the top 16 of a 32k vocabulary, and its normalized entropy stays low through the response.

Clipping and the probability ratio rarely act. Across datasets and models, the PPO loss is clipped on fewer than 5% of tokens per batch, so successive policies stay close and the update is nearly on-policy. Turning clipping off (for the value loss as well), and then also dropping the ratio at $\lambda=1$, which leaves vanilla policy gradientREINFORCE applied per token, with each token's return from that point on minus a learned per-prefix baseline, does not hurt and if anything raises reward slightly. With two gradient steps per batch, as here, there is little room for the policy to move far from the one that sampled the data.

Treating the response as one action, with other samples as the baseline, beats PPO. Since tokens before the last get only the KL part of the reward, and the next state is the current one with the chosen token appended, the problem is a bandita one-step decision problem: one action (here the whole response) and one reward, with no later states with the prompt as context. REINFORCE with a moving-average baseline ties vanilla policy gradient on Pythia and beats it on Llama (55.3% against 52.3% win rate). RLOO beats both, and PPO by 10.3, 14.5 and 32.1 points across the three settings. For two samples it is a contrastive objective weighted by the reward gap,$$\begin{aligned}&\tfrac12\big(R(y^+)-R(y^-)\big)\\&\times\big(\log\pi_\theta(y^+|x)-\log\pi_\theta(y^-|x)\big),\end{aligned}$$maximized, with $y^+$ the higher-reward sample. RAFTreward-ranked fine-tuning: sample $k$ responses, keep the best by reward, and fine-tune on it with ordinary next-token loss, which trains only on the best of the same $k$ samples, averages 56.1% and 59.5% win rates at $k=2$ and 4 against RLOO's 61.3% and 61.9%, and its reward falls more when Gaussian noise of standard deviation 3 or 5 is added to the reward or $\beta$ is raised, since it depends only on which sample ranks first. On helpful-harmless data with Llama 7B, PPO's responses average 16.5 tokens and DPO's 104.4, against about 60 for RLOO.

Limits. Ahmadian et al.'s PPO is one implementation, its learning rate swept over two values, and its responses on helpful-harmless data average 16.5 tokens against 39 to 104 for the other methods, which this note reads as a sign that it settled into a short-answer mode (the paper reports only the length); how much of RLOO's margin survives a PPO built with the care of Huang et al. is not tested. Win rates there come from a GPT-4 judge, models are at most 7B, and reward over-optimization is not studied. Huang et al. study one task with a GPT-3.5 judge, and Ziegler et al.'s results come from few seeds and few labelers per comparison. The case for RLOO assumes a reward only on the full response; with intermediate rewards, per-token credit assignment may matter again.