Tülu 3 post-training recipe

The evaluations are fixed before training, and training prompts are decontaminated by 8-gram overlap. Each development benchmark has one fixed formulation in the released evaluation code, down to the answer extraction, which alone moves scores by more than many effects measured here: on MATH, trying three answer formats in place of one raised reported scores by up to 10 points for models that ignored the examples' format. The unseen suite was designed separately, with prompts written as users write them (no worked examples, an instruction on the answer format), and its scores were not looked at during development. Overlap is measured on prompts onlywhy: completions in training sets are often regenerated by a language model, so the prompt is what links a training example to a test item: a test token matches a training instance if the two share an 8-gram containing it, and a test instance overlaps when more than half of its tokens match one training instance. Partial matching catches near copies such as a math problem with its numbers changed; embedding similarity could not tell a paraphrase from text on the same topic. A training set overlapping more than 2% of any evaluation's instances counted as contaminated; against the unseen suite such sets were dropped, and against the development suite they were dropped if that cost little and otherwise cleaned instance by instance. Popular sets failed the check: the code-instruction set Evol CodeAlpaca overlaps 70.7% of HumanEval, the math set NuminaMath-TIR 18.2% of MATH, and LMSys Chat 1M, a log of real user chats, 46.5% of AlpacaEval.

The SFT mix is assembled one skill at a time, and each targeted source raises its own benchmarks. For each lagging skill the authors first found the data that did best on it alone (an estimate of what the setup could reach), then combined these mixes and added, removed, decontaminated and downsampled sources while tracking the average. Public sets came in for diversity, such as WildChata large public log of real user conversations with chat models; the subset answered by GPT-4 is used, and for specific skills, with licenses traced through each composite dataset. Gaps were filled by persona-driven synthesisprompting GPT-4o to write a problem as a given persona would pose it, such as “a machine learning researcher focused on neural networks”, with about 250k personas to keep the data varied, which produced about 220k math problems, 35k coding problems and 30k prompts with IFEval-style constraints. Removing one source from the final 8B mix lowers its own benchmarks: without the math data GSM8K falls from 76.2 to 64.1 and MATH from 31.5 to 23.5; without the persona data IFEval falls from 72.8 to 53.6; without the safety data the safety average falls from 93.1 to 74.7 while other skills barely move; without WildChat AlpacaEval falls from 12.4 to 7.5. More data helped on average up to the full mix. The base model limits what the mix can do: the same mix gives 56.4 on MATH on Qwen 2.5 Math 7B against 31.5 on Llama 3.1 8B.

Summing the token losses in SFT removes a dependence on how batches are split, and at 70B the seed alone moves the average by 2.6 points. Early SFT runs in the authors' code scored below the same runs elsewhere, and the cause was how the loss was averaged. Take a batch of two sequences, with $n_j$the number of non-padding tokens of sequence $j$ and $\ell_j$the summed next-token loss over the non-padding tokens of sequence $j$. One forward pass on both averages over tokens, while gradient accumulation with one sequence per micro-batch averages the per-sequence means:$$\begin{aligned}\text{one pass:}\quad&\frac{\ell_1+\ell_2}{n_1+n_2},\\ \text{accumulated:}\quad&\frac12\Big(\frac{\ell_1}{n_1}+\frac{\ell_2}{n_2}\Big).\end{aligned}$$The first weights each token equally and the second each sequence equally, so changing the accumulation steps or the number of devices reweights the data. The sum $\ell_1+\ell_2$ gives the same gradient however the batch is split and weights tokens equally, which also scored better on early mixes; the learning rate must be retuned, since the gradient now grows with the tokens per batch. The final runs use learning rates of $5\times10^{-6}$ at 8B and $2\times10^{-6}$ at 70B for two epochs, and more epochs did not help in a sweep. The seed alone moved the average between 59.8 and 60.1 at 8B and between 70.0 and 72.6 at 70B, and a model soupa model whose weights are the average of the weights of several fine-tuned models with the same starting point of two seeds did not reliably beat the best single seed, so the final SFT models are best single runs.

Preference data improves with distinct prompts and with the model's own responses, and the choice of LLM judge hardly matters. The pipeline extends that of UltraFeedbacka public preference dataset of about 64k prompts, each answered by four models and rated by GPT-4. Each prompt gets four responses from models drawn from a pool of 22, and on a share of prompts one response is the SFT model's own (on-policygenerated by the model that the stage is about to train, here the SFT model data). GPT-4o rates each from 1 to 5 on helpfulness, instruction following, honesty and truthfulness; the response with the best mean rating is chosen and a random lower-rated one rejected. In ablations at 8B, more distinct prompts raised the scores, while expanding 64k UltraFeedback prompts to 383k pairs by re-pairing each prompt's four responses gave no gain on average. On the same prompts, including the SFT model's responses beat a fully off-policy set. Prompts unused in SFT did slightly better than reused ones, and a mix of both did best. Five judges, from GPT-4o to Llama 3.1 70B, rating the same 10k prompts gave DPO averages between 56.6 and 57.3. The final mix beat DPO on UltraFeedback by 1.8 points at 8B and 3.3 at 70B. Of three persona preference sets, only instruction following helped; in its pairs the rejected response answers a rewrite of the prompt with one constraint relaxed, so it fails the original constraint.

Length-normalized DPO was the only preference loss that beat its starting model, and PPO came close to it at about 14 times the compute. DPO's implicit reward for a response, $\beta\log(\htmlData{gr=g15}{\pi_\theta}(y|x)/\pi_{\rm ref}(y|x))$, is a sum over tokens, so its size grows with length, and judges tend to prefer longer responses. Length-normalized DPO divides each log ratio by the length $|y|$the number of tokens in response $y$. With $y_c$the chosen response of a preference pair; $y_r$ is the rejected one and $y_r$ the rejected one, and $\sigma$the logistic function $\sigma(z)=1/(1+e^{-z})$ the logistic function, it minimizes$$\begin{aligned}\mathcal L&=-\mathbb E\,\log\sigma\big(m(y_c)-m(y_r)\big),\\ m(y)&=\underbrace{\frac{\htmlData{gr=g19}{\beta}}{|y|}\log\frac{\pi_\theta(y|x)}{\pi_{\rm ref}(y|x)}}_{\beta\ \times\ \text{log ratio per token}},\end{aligned}$$over preference triples $(x,y_c,y_r)$, with $\pi_{\rm ref}$ the SFT model. On UltraFeedback from an early SFT checkpoint averaging 55.7, the best run of each loss scored 57.3 for length-normalized DPO, 55.2 for DPO, 52.9 for SimPOa preference loss on length-normalized log-probabilities with no reference model and a fixed target margin between chosen and rejected and 55.5 for PPO against a reward model. The final setting is $\beta=5$, one epoch, batch 128 and a learning rate of $5\times10^{-7}$ at 8B ($2\times10^{-7}$ at 70B); $\beta$ is large next to the 0.1 of the plain DPO run presumably because it multiplies the log ratio divided by a length of hundreds of tokens. The result was sensitive to the settings: with three epochs, $\beta=2$ scored 46.8 against 53.4 at $\beta=5$, and one epoch in place of three gave 57.3. In a later comparison on the same prompts, PPO with a reward model trained on that mix averaged slightly below DPO and took 28 hours on two nodes against 4 on one, so PPO was kept for RLVR alone.

RLVR raises the checked skills, a reward model is the best start for the value model, and adding its score to the reward does not help. The PPO setup follows earlier open PPO work: the value model starts from a reward model trained on general preference data, dropout is offwhy: PPO's ratio compares a token's probability at learning with its probability at sampling; dropout makes the two differ before any update, so ratios can be clipped and give no gradient, and the KL estimate is noisy, prompts are reshuffled each epoch, a response cut off without an end-of-sequence token gets reward $-10$why: to push the policy to finish its responses within the length limit, and advantages are standardized. In ablations from an intermediate DPO model with $\beta$ between 0.01 and 0.1, training on each of GSM8K, MATH and the constraint prompts raised the training reward and the matching test score, while the average over all benchmarks did not always rise. Initializing the value model from the general reward model gave a better GSM8K score and average than initializing it from the DPO policy. Adding the reward model's score to the checker's did worse on GSM8K and made the average noisier. Starting from the weaker SFT model reached the same training reward at a larger KL, and the stronger start usually tested better. As $\beta$ fell and the KL grew, the average generally fell; on the constraint prompts at $\beta=0.01$, asked for a pen's length in centimeters with the keywords “timberlike” and “unconnectedly” included, the policy answered “timberlike, unconnectedly”, which passes the checker and says nothing about the pen; at $\beta=0.1$ it estimated 15 to 20 cm.

The final RLVR runs choose checkpoints on the development suite, and their gains shrink where a benchmark is near its ceiling. The final runs start from the best DPO models and train on all three prompt sources together. Checkpoints were scored every 100 steps (40 at 70B) and at 8B the one best on MATH and IFEval kept; some 8B runs reached 89.4 on GSM8K and 84.8 on IFEval but did worse elsewhere. At 70B, GSM8K started at 93.5 and did not move, and the KL stayed below 1, which the authors attribute to the learning rate of $10^{-7}$ (a higher one made the KL explode early). At 405B, GSM8K was saturated after DPO and the constraint prompts did not help early on, so RLVR used only MATH, which rose more than 5 points within 25 steps and was still rising when compute ran out at step 75. Generation and training run asynchronously: vLLMan open inference engine that serves many generations at once by storing the attention key-value cache in pages generates on dedicated GPUs while the learners update, the policy trains on the second-latest batch of generationswhy: when generation runs ahead of training, how stale the samples are varies from step to step; fixing the lag at one update keeps runs reproducible, and new weights are broadcast after each update. At 405B an iteration took about 550 seconds of generation and 1,500 of training, with an 8B value model to save compute.

The unseen suite shows which development choices generalized. Each skill pairs a development and an unseen benchmark: MMLU with GPQAvery hard multiple-choice questions in biology, physics and chemistry written by domain experts, BigBenchHard with AGIEvalmultiple-choice questions from admission and qualification exams such as the SAT and LSAT; the English subset is used, MATH with DeepMind Mathematicsgenerated questions in 56 categories of arithmetic, algebra and calculus, with answers compared symbolically, HumanEval with BigCodeBenchharder Python tasks that call common libraries, scored by unit tests; its hard subset of 148 tasks is used, and IFEval with IFEval-OODa new benchmark built like IFEval on unseen WildChat prompts, with 52 constraint types disjoint from IFEval's 25. Averaged over these five pairs, the final models score best on both sides: at 8B, 64.9 on the development benchmarks and 29.9 on the unseen ones after SFT against 68.8 and 32.4 at the end; at 70B, 78.1 and 41.0 against 80.7 and 44.4. Without the math data MATH falls 8.0 points and DeepMind Mathematics 9.0, so that data carried over. Without the persona data IFEval falls from 72.8 to 53.6 while IFEval-OOD moves from 17.6 to 18.0, so that data did not carry over. Every model tested scores far lower on IFEval-OOD than on IFEval (Tülu 3 8B 24.3 against 82.4, Llama 3.1 8B Instruct 26.1 against 80.6), which the authors read as models fitting IFEval's own constraint types. More DPO data raised MATH more than DeepMind Mathematics, which the authors trace to the models writing LaTeX where DeepMind Mathematics does not expect it, which broke answer extraction.

Online DPO did not help, and rejection sampling gained little for its compute. Online DPODPO on pairs sampled from the current policy during training and labeled by a reward model or judge, in place of a fixed dataset, run for 200k episodes on math prompts with a reward model further trained on math preferences, gave little or no gain on GSM8K and lowered MATH across the temperatures and KL coefficients tried. Rejection samplinggenerating several responses per SFT prompt with the trained model, keeping the one a reward model or judge ranks best, and rerunning the pipeline on the new data gave small gains for its compute; open judges struggled to pick the best candidate, and including the original response among the candidates worked much better than choosing among new ones alone.