Chinchilla scaling

The envelope of matched-schedule training curves scales as $C^{0.50}$. For each size, train four runs whose cosine cycles span a 16-fold range of horizons, and at each FLOP value record which run reaches the lowest loss and at which $(N,D)$; power-law fits give $a=b=0.50$. Every selected point lies in the last 15% of its run, which gives the practical rule of ending a run where its cosine cycle ends. A loss read mid-schedule, before the learning rate has decayed, overstates what that $(N,D)$ can reach, and cycles that overshoot the run by more than 25% clearly hurt.

IsoFLOP profiles have a clear valley whose minimizer moves as $C^{0.49}$. At nine budgets from $6\times10^{18}$ to $3\times10^{21}$ FLOPs, vary $N$ with $D$ set by the budget and the schedule matched to $D$, and fit a parabola in $\log N$ to each profile. Every profile has a minimum, and the minimizers follow $N_{opt}\propto C^{0.49}$, $D_{opt}\propto C^{0.51}$. This is the most direct estimator, though its largest budget is about 200 times smaller than Gopher's.

The parametric fit, once correctly optimized, agrees with the other two. Fit $\hat L(N,D)=E+A/N^\alpha+B/D^\beta$ (entropy of text, finite model, finite one-pass training) to all final losses with a Huber loss on $\log L$ and L-BFGS. Under $6ND=C$ the optimum is $$N_{opt}=G\Big(\frac C6\Big)^{\frac{\beta}{\alpha+\beta}},\qquad D_{opt}=G^{-1}\Big(\frac C6\Big)^{\frac{\alpha}{\alpha+\beta}},\qquad G=\Big(\frac{\alpha A}{\beta B}\Big)^{\frac1{\alpha+\beta}}.$$ As published, the fit gives $a=0.46$ and a 40B optimum at Gopher's budget. Besiroglu et al. (2024), refitting 240 runs digitized from the paper's contour plot, find a fitting error instead: averaging instead of summing the Huber terms made L-BFGS stop early, which the authors confirmed and which explains the implausibly narrow bootstrap interval $a\in[0.454,0.455]$. Their refit, $E=1.82$, $A=482$, $B=2085$, $\alpha=0.348$, $\beta=0.366$, gives $a=0.51$ with standard error about 0.02. At Gopher's budget (my check) the published fit gives about 40B parameters and 59 tokens per parameter, the refit about 72B and 18. So, if the digitization is faithful, the corrected Approach 3 agrees with Steps 1 and 2 and with how Chinchilla was trained, and the published Approach 3 numbers and their prediction of even smaller models do not stand.

Why Kaplan's allocation differs is not settled by this paper. Kaplan trained every model on one fixed-length cosine schedule and read compute-optimal points off intermediate steps, whose mid-schedule losses, the paper argues, undervalue shorter training and push the fit toward larger models. And many of Kaplan's runs are under 100M parameters while the frontier $\log N_{opt}$ is concave in $\log C$, so slopes fit on small models overpredict size at large budgets. In the one head-to-head test, at $10^{21}$ FLOPs, the 2.80B model chosen by Approach 1 beats the 4.74B model chosen by Kaplan's law, which confirms the allocation without isolating a cause. Pearce and Song (2024; read here from the abstract only) attribute much of the gap to Kaplan counting non-embedding parameters at small scale. The mechanism is our gloss, checked only by a toy calculation: embeddings are a large share of a small model and a shrinking share of a large one, so along the same frontier the non-embedding count grows faster than the total and its fitted exponent comes out above 0.5. Porian et al. (2024; also abstract only) remove Kaplan's exponent by correcting last-layer compute, warmup and optimizer tuning, and report that careful learning-rate decay is not essential, so the schedule explanation remains an untested hypothesis.

At equal compute, the smaller model trained longer is better almost everywhere. Chinchilla also swaps Adam for AdamW, drops NFKC normalization from the tokenizer and keeps float32 weight copies in the optimizer, so the comparison mixes allocation with these changes. It lowers bits per byte on all Pile subsets, raises MMLU from 60.0% to 67.6% and BIG-bench from 54.4% to 65.1%, at about a quarter of Gopher's inference compute per token. The language-modelling gains carry a leakage caveat, since Chinchilla sees four times more data, so the paper leans on MMLU and BIG-bench. One pair of large runs confirms the direction of the prediction and says nothing about the exponent.