Explaining Neural Scaling Laws

Theorem 1 (variance-limited). Let $f_T$ be the network output after $T$ steps, random through initialization, training set and seed, $\ell(f)$ the test loss as a function of the output, and suppose $\mathbb E[(f_T-\mathbb Ef_T)^k]=O(\epsilon)$ for all $k\ge2$. If $\ell$ is a polynomial, has bounded second derivative, or is 2-Hölder, then $\mathbb E\,\ell(f_T)-\ell(\mathbb Ef_T)=O(\epsilon)$. MSE and cross-entropy fall under the second case.

Variance-limited scaling is a second-order Taylor effect. Expand the loss about the mean output, $$\ell(f_T)-\ell(\mathbb Ef_T)=\ell'(\mathbb Ef_T)(f_T-\mathbb Ef_T)+\tfrac12\ell''(c)(f_T-\mathbb Ef_T)^2;$$ the linear term has mean zero, so the excess is the output variance times a bounded curvature. The supplement proves the variance is $O(1/D)$ for GD and SGD on losses polynomial in the weights, for a step count $T$ that does not grow with $D$: each step maps sample means to polynomials in sample means, whose centered moments are $O(1/D)$.

In the linear model the $1/D$ and $1/P$ corrections are explicit. Perturb the sample second moment, $\bar C=C+\delta C$ with $\mathbb E\delta C=0$ and fluctuations of order $1/D$, and expand the inverse in the exact loss: $$L(D,P)=L(P)+\tfrac1D(\text{quartic contraction})+O(D^{-2}).$$ The same expansion in $\delta K$, of order $1/P$ over feature draws, gives the overparameterized case. For centered Gaussian features the coefficient has a closed form (our gloss): the part of the teacher outside the student's span is independent of the student features and acts as label noise, so $L(D,P)-L(P)=L(P)\,P/(D-P-1)$ (to leading order, the $1/(1-N/T)$ inflation in the note on spectrum-dependent learning curves). The expansion fails as $D\to P$, where $\bar C^{-1}$ blows up: that is the interpolation peak, and the reason the regime needs $D\gg P$.

Resolution-limited bounds come from nearest-neighbor distances. On a $d$-dimensional manifold a test point's nearest training point $\hat x$ is at mean distance $O(D^{-1/d})$, and chaining the Lipschitz constants through $f(\hat x)=F(\hat x)$ turns that distance into a loss: $$\ell(f(x))\le K_L|f(x)-F(x)|\le K_L(K_f+K_F)|x-\hat x|.$$ Averaging gives Theorem 2; Theorem 3 reruns it with the $P$ points where the model matches the target. The sharper estimate expands the loss around $\hat x$, where it vanishes to order $n\ge2$, so the leading term is $|x-\hat x|^n\propto D^{-n/d}$. With MSE and a model that interpolates linearly between training points, the error in $f$ is quadratic in the distance and MSE squares it, so $n=4$ and $\alpha\approx4/d$. The bound needs only Lipschitz control; the estimate also assumes smoothness and that the trained model interpolates linearly.

Both exact losses are residuals after a projection, which gives the duality. $L(P)$ is the Schur complement of $C$ after a random rank-$P$ sketch $\mathcal P$, the teacher variance that $P$ random combinations of its features miss. $L(D)$ is the expected kriging variance of a Gaussian process with covariance $K$ after noiseless observations at the $D$ training points. The formulas exchange projecting on features with projecting on data points, and the kernel's integral operator under $p(x)$ has the same nonzero eigenvalues as $C/S$, so one spectrum governs both. The paper takes this as the reason $\alpha_P=\alpha_D$.

A power-law spectrum gives $D^{-\alpha_K}$. In the eigenbasis $e_i$ of $C$, with $\bar e_j$ the eigenvectors of the sample second moment, the overparameterized loss is (up to the $1/S$ normalization) the population variance outside the span of the sample, $$L(D)=\frac12\sum_i\lambda_i\Big(1-\sum_{j\le D}(e_i\cdot\bar e_j)^2\Big).$$ For large $\alpha_K$ the sample eigenvectors are close to the population ones, so the span is the top $D$ directions and $L(D)\propto\sum_{i\gt D}i^{-(1+\alpha_K)}\approx D^{-\alpha_K}/\alpha_K$. For general $\alpha_K$ the paper plugs the spectrum into Canatar et al.'s ridgeless, noiseless learning curve, the $\kappa$ equation of Bordelon et al., and finds $\kappa\propto D^{-\alpha_K}$, $\gamma\propto D^0$, $L\propto D^{-\alpha_K}$. In Bordelon's notation an isotropic teacher puts power $\lambda_i/S$ on mode $i$, so $b=a=1+\alpha_K$ and $\min\{a-1,2b\}=\alpha_K$: the unlearned tail always dominates. $L(P)$ follows by duality.

Smoothness of the kernel ties the exponent to the manifold dimension. On the $d$-torus, Kühn's bound $\lambda_n\lesssim n^{-(1+t/d)}$ for a $C^t$ kernel is Fourier counting: a translation-invariant kernel's eigenvalues are its Fourier coefficients $b_{n_I}\ge0$, and $C^t$ smoothness ($t$ even) keeps $\sum_I|n_I|^tb_{n_I}$ finite; at most about $N/2$ frequencies have $|n_I|^d\lesssim N$, so at least half of the $N$ largest coefficients sit at $|n_I|^t\gtrsim N^{t/d}$, and the $N$-th largest eigenvalue is at most of order $N^{-(1+t/d)}$. A feature covariance that is such a kernel on the data manifold and saturates the bound has $\alpha_K=t/d$, the spectral version of Step 3's $n/d$ estimate; saturation is an assumption.

Measured exponents are near 1 in the variance-limited regime and track the data's intrinsic dimension in the resolution-limited one. Variance-limited exponents come out near 1 everywhere: $\alpha_D$ from 0.98 to 1.11 for small networks on image datasets with $D\gg P$, and $\alpha_W$ from 0.98 to 1.03 in width. Resolution-limited ones depend on the data: they follow $4/d$ for teacher-student networks on a $d$-dimensional cube, while on real data the relation to an estimated intrinsic dimension is, in the paper's words, less clear. For random ReLU features on pooled MNIST, $\alpha_D$ and $\alpha_P$ roughly track the fitted $\alpha_K$. Grouping CIFAR-100 into 5 to 100 superclasses leaves $\alpha_D\approx0.4$, while Gaussian input noise lowers it from 0.58 to 0.29.