Direct preference optimization

The KL penalty turns reward maximization into fitting a fixed target. Without it, $\mathbb E_\pi r$ is linear in $\pi$, so its maximum over the simplex is a vertex, all mass on the best sequence; even a reward that ranks every sequence correctly asks for a model that emits one string. Folding the penalty into the reward, as PPO does per token, gives $r'_\theta=r+\beta(\log\pi_{\rm ref}-\log\pi_\theta)$, a reward that moves with the policy and whose $-\beta\log\pi_\theta$ part penalizes concentrating mass, so this is no longer reward maximization in the usual sense. Instead combine the two terms under one logarithm, $-\beta\,\mathbb E_\pi\log[\pi/(\pi_{\rm ref}e^{r/\beta})]$, and insert $Z$: that is (a), the Gibbs variational principle, so the regularized problem is minimizing a reverse KL to a target that does not depend on $\pi$. The target puts mass exactly where $\pi_{\rm ref}$ does; it returns $\pi_{\rm ref}$ as $\beta\to\infty$ and concentrates on the best sequences in that support as $\beta\to0$, so collapse is the $\beta=0$ end of the same family.

The tilt is a posterior, which splits alignment into choosing a target and approximating it. If $r\le0$ (shifting $r$ changes $Z$ and not $\pi^*$), introduce an event $O$ with $p(O=1\mid y)=e^{r(y)}$ and absorb $\beta$ into $r$; Bayes' rule gives $p(y\mid O=1)=\pi^*$ with $Z=p(O=1)$, and Jensen gives $$\log p(O=1)\ \ge\ \mathbb E_{\pi_\theta}r-\mathrm{KL}(\pi_\theta\,\|\,\pi_{\rm ref}),$$ with gap $\mathrm{KL}(\pi_\theta\|\pi^*)$, so KL-regularized RL is variational inference with $\pi_\theta$ as the family and $\beta$ as the temperature of the tilt. The posterior is formal, since "this sample is optimal" is a modelling device. The split separates failure modes: a wrong target (the exponential tilt is one choice; a hard constraint gives $\pi_{\rm ref}$ renormalized on the allowed set) is a modelling error, a poor fit is an inference error, and decoding-time methods are the sampling-based alternative to fine-tuning. Korbak et al. then argue, without a theorem, that the divergence is responsible for the success: RL without the KL term degenerates, while divergence minimization without an RL form (forward KL to the same target, or maximum likelihood) works.

DPO solves the tilt for the reward, and the normalizer cancels in a comparison. Take logs of the closed form: $$r(x,y)=\beta\log\frac{\pi_r(y|x)}{\pi_{\rm ref}(y|x)}+\beta\log Z(x).$$ $Z(x)$ sums over all responses, which is why RLHF approximates $\pi_r$ with PPO. But Bradley-Terry compares two responses to the same prompt, both carry the same $\beta\log Z(x)$, and it cancels, so the preference likelihood is a function of the policy; parametrize the policy in place of the reward and maximize it, and that is $\mathcal L_{\rm DPO}$ (Plackett-Luce rankings cancel the same way). The fitted policy is KL-optimal for its implicit reward by construction, so reward fitting and policy optimization are one step, needing only log-probabilities of two responses under $\pi_\theta$ and the frozen $\pi_{\rm ref}$.

Restricting to implicit rewards loses no generality. Classes are the right notion because both ends of the pipeline ignore prompt-only terms: a shift $f(x)$ cancels in every Bradley-Terry or Plackett-Luce factor and in the tilt's normalizer. For existence, the projection $r\mapsto r-\beta\log\sum_y\pi_{\rm ref}(y|x)e^{r(x,y)/\beta}$ stays in the class and, by Step 3, equals $\beta\log\pi_r/\pi_{\rm ref}$. For uniqueness, if two implicit rewards differ by $f(x)$, then $\pi'=\pi e^{f/\beta}$, and summing over $y$ forces $f=0$; positivity of $\pi_{\rm ref}$ keeps every log ratio finite. So the implicit reward is an identifiability constraint, the representative whose partition function is 1 for every prompt. The same normalizer is the soft value of $\pi_{\rm ref}$; left in the reward, it is a prompt-dependent offset that inflates the variance of the policy gradient, which PPO's learned value function estimates as a baseline, and the implicit reward has it removed, so DPO needs none (the paper's argument, not a measurement).

The gradient is a weighted likelihood-ratio update, and the weight prevents degeneration. $$\nabla_\theta\mathcal L_{\rm DPO}=-\beta\,\mathbb E\Big[\sigma\big(\hat r_\theta(x,y_l)-\hat r_\theta(x,y_w)\big)\big(\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)\big)\Big].$$ The update raises $\log\pi_\theta(y_w|x)$ and lowers $\log\pi_\theta(y_l|x)$, weighted by the probability the implicit reward model assigns to the wrong ordering, so pairs already ordered with a wide margin stop contributing. Drop the weight and the update is unlikelihood training, maximize $\log\pi(y_w|x)-\alpha\log\pi(y_l|x)$, which is unbounded because $\log\pi(y_l|x)$ can be pushed to $-\infty$; on summarization it produces strings like "when when when", the degeneration the weighting prevents.

DPO optimizes the shared objective better than PPO and does at least as well on real tasks. On IMDb, with a sentiment classifier as the true reward, DPO's reward-KL frontier dominates PPO's, even PPO trained on the true reward. On Reddit TL;DR with a 6B model, GPT-4 prefers DPO's summaries to the reference about 61% of the time, against 57% for PPO at its best temperature, and DPO degrades less as temperature rises; on HH dialogue it roughly matches best-of-128. The caveats are mostly the paper's own: models up to 6B, $\beta$ barely tuned (0.1, and 0.5 for TL;DR), GPT-4 as the main judge, and PPO baselines whose quality depends on implementation.

On near-deterministic preferences DPO's loss ignores $\beta$. If $p^*(y\succ y')=1$, Bradley-Terry needs $r(y)-r(y')=+\infty$, and the tilt sets $\pi^*(y')=0$ for every $\beta$. The finite-sample version matters in practice: a pair seen once has empirical preference 1 even when $p^*=0.8$. There the DPO loss of the margin is $-\log\sigma(\beta h)$, strictly decreasing with no finite minimizer, and $\beta$ only rescales $h$, so the optimum is $h=\infty$ and $\pi_\theta(y_l|x)/\pi_{\rm ref}(y_l|x)\to0$ whatever $\beta$ is. This is logistic regression on separable data (Soudry et al., 2018): the margin grows slowly without bound, and only early stopping limits it. The loss also sees the policy only through log-ratio differences on observed pairs, so the probability of responses outside the data is unconstrained. Neither gap is discussed in the DPO paper. Azar et al. argue, without testing it, that RLHF suffers less because its underfit reward model keeps rewards finite on {0,1} data.

IPO bounds the tilt by using the win rate itself, and its optimum solves linear equations. Azar et al. write all three methods as one objective, ΨPO: maximize $\mathbb E_{y\sim\pi,\,y'\sim\mu}\Psi\big(p^*(y\succ y')\big)-\tau\,\mathrm{KL}(\pi\|\pi_{\rm ref})$ for a non-decreasing $\Psi$, with nothing tying $p^*$ to a reward. By the same Gibbs argument the optimum is $\pi_{\rm ref}e^{g/\tau}$ normalized, with $g(y)=\mathbb E_{y'\sim\mu}\Psi(p^*(y\succ y'))$, so $\log(\pi^*/\pi_{\rm ref})$ moves exactly as far as $g/\tau$. With the logit, $\Psi(\sigma(r(y)-r(y')))=r(y)-r(y')$ and $g$ is the RLHF reward up to a constant, which proves that RLHF and DPO are instances of ΨPO; the logit is unbounded, which is the problem of Step 7. With the identity, $g(y)=p^*(y\succ\mu)\in[0,1]$, so log ratios span at most $1/\tau$ however deterministic the preferences are. Log ratios of the tilt remove the normalizer and leave equations linear in the logits $s=\log\pi$, $$h_{\pi^*}(y,y')=\tau^{-1}\big(p^*(y\succ\mu)-p^*(y'\succ\mu)\big)\qquad\text{for all }y,y',$$ and the IPO loss is their mean squared residual under $\mu\otimes\mu$, so $L\ge0=L(\pi^*)$. In the logits it is a convex quadratic whose second-order part $\sum\mu(y)\mu(y')(s(y)-s(y'))^2$ vanishes only along the all-ones direction, which leaves the softmax unchanged, so the minimizer is unique as a policy and every local minimum is it. Without the support condition the quadratic does not see actions $\mu$ never samples, and their mass is left free.

The sampled loss is unbiased, and it regresses the margin to $1/(2\tau)$. The Bernoulli label can replace the win-rate difference although the two targets differ pointwise, because the loss sees the target only through a cross term with the margin, and $h_\pi(y,y')=f(y)-f(y')$ with $f=\log(\pi/\pi_{\rm ref})$, so that term needs only the label's one-sided means, $\mathbb E[I\mid y]=p^*(y\succ\mu)$ and $\mathbb E[I\mid y']=1-p^*(y'\succ\mu)$. On a dataset each comparison enters as $(y_w,y_l,1)$ and $(y_l,y_w,0)$, and with $h$ antisymmetric it contributes $$\tfrac12\big[(h-\tau^{-1})^2+h^2\big]=\big(h-\tfrac1{2\tau}\big)^2+\text{const}.$$ DPO and IPO act on the same margin and differ only in the loss: the logistic loss always decreases as the margin grows, while the square has a finite target, and that target provides the regularization.

In small bandits DPO goes greedy or drops actions for every $\tau$, while IPO stays a $\tau$-controlled tilt. With two actions, uniform $\pi_{\rm ref}=\mu$ and $p^*(y_1\succ y_2)=1$, IPO gives $\pi^*(y_1)=\sigma(1/(2\tau))$ and DPO gives 1 for every $\tau$. With three actions and a total order seen once per pair, DPO converges to the top action for $\tau\in\{0.1,0.5,1\}$, while IPO gives equal log gaps of $1/(3\tau)$ (my computation), $(0.45,0.32,0.23)$ at $\tau=1$. The commoner failure in a large action space with little data is an action that never wins: when $y_a$ and $y_b$ each beat the other once and $y_c$ loses once, DPO sends $y_c$ to 0, and IPO leaves it 0.16 and 0.23 at $\tau=0.5$ and 1.

Consequences and limits. DPO turned preference fine-tuning into supervised learning on a fixed dataset. DPO, IPO and RLHF share one target, the tilt $\pi_{\rm ref}e^{g/\tau}$, and differ in the tilting function and in what stops the fit from running to the boundary: early stopping for DPO, an underfit reward model for RLHF, a finite target margin for IPO. A drawback of IPO is that its target is a win rate against the sampling policy $\mu$, so its optimum depends on how the pairs were generated, whereas under Bradley-Terry the logit turns $\mu$ into a constant shift of $g$; and its evidence comes only from toy bandits, with no language-model experiments. The tilt also says how reward errors enter (not discussed by Korbak et al.): with a learned $r=r_{\rm true}+e$ the target gains a factor $e^{e/\beta}$, so errors are amplified without bound as $\beta$ shrinks, one mechanism for the over-optimization Gao et al. measure.