InstructGPT

Context. Next-token prediction on web text differs from "do what this user asks, truthfully and without harm", and scale alone does not close the gap: a larger GPT-3 still ignores instructions, makes things up and needs careful prompting. Learning a reward from human comparisons and optimizing it with RL had worked on control tasks (Christiano et al., 2017) and summarization (Stiennon et al., 2020). This paper runs the recipe on the open-ended prompts API users actually send, at 1.3B, 6B and 175B, and asks what each stage contributes and what it costs. A reward model fit to comparisons is a cheap way to state what labelers want. Optimizing it from a supervised start, with a KL penalty that keeps the policy near that start, is worth more than a hundredfold increase in parameters. The penalty guards against exploiting the reward model, and keeping general capabilities requires the pretraining data itself.

Setting. GPT-3 is trained in three stages on three kinds of labeler data, with test prompts from held-out customers. Supervised fine-tuning (SFT) on demonstrations for about 13k prompts runs for 16 epochs, well past where validation loss overfits, because reward-model score and human ratings keep improving. The reward model, a 6B model with a scalar head, is fit to rankings of $K=4$ to $9$ outputs per prompt (33k prompts) by Bradley-Terry maximum likelihood, with all $\binom K2$ pairs of a prompt in one batch element: $\mathcal L(\theta)=-\tbinom K2^{-1}\,\mathbb E_{(x,y_w,y_l)}\log\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big),$ then shifted so demonstrations score 0 on average. PPO on 31k API prompts maximizes $\mathbb E_{x\sim D,\;y\sim\pi_\phi(\cdot|x)}\Big[r_\theta(x,y)-\beta\log\frac{\pi_\phi(y|x)}{\pi^{\rm SFT}(y|x)}\Big]+\gamma\,\mathbb E_{x\sim D_{\rm pretrain}}\log\pi_\phi(x),$ the learned reward with a per-token KL penalty to the supervised start ( $\beta=0.02$ ) plus a pull toward the pretraining distribution. "PPO" has $\gamma=0$ ; "PPO-ptx", the model called InstructGPT, has $\gamma=27.8$ . The reference $\pi^{\rm SFT}$ is a 2-epoch SFT model with 10% pretraining data mixed in, distinct from the 16-epoch SFT baseline. The main metric is how often labelers prefer a model's output to the 175B SFT model's.

Main findings (all empirical). Labelers prefer InstructGPT at every size: 175B InstructGPT beats 175B GPT-3 85 $\pm$ 3% of the time and few-shot GPT-3 71 $\pm$ 4%, and the 1.3B InstructGPT is preferred to the 175B GPT-3. The preferences carry over to labelers who produced no training data. Plain PPO loses ground on public NLP benchmarks (the "alignment tax"); mixing in pretraining gradients removes most of the loss without lowering preference scores, and raising the KL coefficient does not. The cost is small: 60 petaflop/s-days for the 175B PPO-ptx model and 4.9 for its SFT stage, against 3,640 to pretrain GPT-3.

Step 1

Each stage adds preference, and prompting does not substitute for training. At every size, win rates rank GPT-3 lowest, then GPT-3 with a tuned few-shot instruction prefix, SFT, and PPO, with PPO-ptx close to PPO, so demonstrations beat prompt engineering and comparisons add on top of demonstrations. The picture is the same on prompts written for plain GPT-3. InstructGPT wins by attempting the right task more often, following explicit constraints such as length, and making up information on closed-domain tasks about half as often (21% against 41%).

Step 2

The gains are not an artifact of fitting the training labelers. Held-out labelers, who wrote no training data, prefer InstructGPT at about the same rates. A reward model trained on four of five labeler groups scores 69.6 $\pm$ 0.9% on the held-out group against 72.4 $\pm$ 0.4% within the training groups, and two training labelers agree only 72.6 $\pm$ 1.5% of the time, so the reward model predicts a new labeler about as well as another labeler does. This rules out individual idiosyncrasies and leaves open everything the labelers share: the researchers' instructions and the customer prompt distribution.

Step 3

The alignment tax comes from losing pretraining behavior, and only pretraining data restores it. PPO regresses against GPT-3 on SQuADv2, DROP, HellaSwag and WMT French-to-English, mostly few-shot. The $\gamma$ term recovers most of this and surpasses GPT-3 on HellaSwag, though DROP, SQuADv2 and translation still lag. On 1.3B, with GPT-3 as the KL reference (similar results with SFT), raising $\beta$ to 2.0 with $\gamma=0$ never repairs DROP and SQuAD and lowers validation reward substantially, while $\gamma\ge20$ repairs them at a small cost in reward. One reading, mine and not the paper's: the KL term is evaluated only on RL prompts, while the lost skills are exercised on other inputs, which only the pretraining likelihood reaches. The mix can also bring back undesirable behaviors from the pretraining data.

On the 1.3B model with

\gamma=0

, raising

\beta

to 2.0 does not repair DROP and SQuAD, while

\gamma\ge20

does. What does this indicate?

Step 4

The KL term exists to keep the policy where the reward model can be trusted. The stated purpose of the penalty is to mitigate over-optimization of the reward model, learned reward rising while true preference falls. The per-token form is the sequence penalty, since an autoregressive log-probability is a sum over tokens and the per-token log ratios add to $\log\pi_\phi(y|x)/\pi^{\rm SFT}(y|x)$ , whose expectation is the KL. The only direct evidence is a sweep of Likert scores against $\beta$ for PPO-ptx: $\beta=0$ and $\beta=2$ both do poorly and the best values are 0.01 to 0.02. The paper never plots learned reward against true preference, which Gao et al. later measure.

Step 5

Public instruction datasets do not match what users ask for. GPT-3 175B fine-tuned on FLAN or T0++ ties prompted GPT-3 and falls below SFT, and InstructGPT is preferred 78 $\pm$ 4% of the time over the FLAN model and 79 $\pm$ 4% over T0. Classification and QA, which public datasets cover because they are easy to score, make up about 18% of API prompts, while open-ended generation and brainstorming make up about 57%, so this compares datasets as much as methods.

Step 6

Safety metrics improve a little, and instruction following also applies to harmful requests. The model follows the prompt in either direction. Toxicity drops by about 25% only when the prompt asks for a respectful answer, and asked to be toxic InstructGPT is more toxic than GPT-3. Truthfulness improves a little on TruthfulQA, and told to say "I have no comment" when unsure, the model more often declines to answer than states a confident falsehood. Bias on Winogender and CrowS-Pairs does not improve. Instruction following generalizes to other languages and code, both rare in the fine-tuning set. The main failures are accepting false premises and hedging on simple questions, which the authors attribute to labelers rewarding epistemic humility.

Consequences and limits. SFT, a Bradley-Terry reward model and KL-penalized PPO became the standard recipe, and the cost figures made the case that, for this prompt distribution, human-feedback fine-tuning is a far cheaper improvement than scale. What it aligns to is narrow and stated plainly: about 40 contractors, mostly in the US and Southeast Asia, following the researchers' instructions, on prompts from API customers, with most comparisons labeled once. The model does what it is asked, including harmful things. On the method side, one 6B reward model serves policies of every size, $\beta$ is tuned by human evaluation alone, and over-optimization is assumed, never measured.

Terms

Bradley-Terry: $P(y_1\succ y_2\mid x)=\sigma\big(r(x,y_1)-r(x,y_2)\big)$ ; the reward is identified only up to a function of $x$
$\binom K2$ per prompt: all pairs from one ranking go in one batch element; shuffled separately, the reward model overfits in one pass
$\pi^{\rm SFT}$: supervised start of RL and the KL reference; a 2-epoch SFT with 10% pretraining mix
$\beta=0.02$: per-token KL coefficient; summed over tokens it is the sequence KL to $\pi^{\rm SFT}$
PPO-ptx, $\gamma=27.8$: PPO plus pretraining log-likelihood gradients; the model called InstructGPT
Alignment tax: regression on public benchmarks (SQuADv2, DROP, HellaSwag, WMT Fr-En) caused by RLHF
Win rate vs 175B SFT: fraction of held-out-customer prompts on which labelers prefer the model's output
Held-out labelers: labelers who wrote no training data; similar win rates, and RM accuracy 69.6% vs 72.4%
Reward over-optimization: learned reward rises while true preference falls; what the KL term is meant to prevent
Compute: 60 petaflop/s-days for 175B PPO-ptx, 4.9 for SFT, 3,640 for GPT-3 pretraining

Quiz

The reward model trains on all $\binom K2$ comparisons from a ranking as a single batch element. What problem does this solve?
Before RL the reward model is shifted by a constant so that demonstrations score 0 on average. Why does this leave the problem unchanged?
PPO subtracts $\beta\log\pi_\phi/\pi^{\rm SFT}$ at every token. Why is this the same as the sequence-level penalty in the objective?
At every size, labelers rank GPT-3 below GPT-3 with a tuned few-shot prefix, then SFT, then PPO. What does this ordering tell a practitioner about the stages?

Sources

Long Ouyang et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, 2017.
Nisan Stiennon et al. Learning to summarize from human feedback. NeurIPS, 2020.