Spectrum-dependent learning curves

Main result: the self-consistent learning curve. A kernel has Mercer eigenvalues $\eta_\rho$ under the data distribution, the target $\bar f=\sum_\rho\bar w_\rho\psi_\rho$ ($\psi_\rho=\sqrt{\eta_\rho}\phi_\rho$) puts power $\eta_\rho\bar w_\rho^2$ on mode $\rho$, labels carry noise variance $\sigma^2$, and kernel ridge uses a penalty $\lambda$ not divided by $P$ ($\lambda\to0$ is the interpolant, NTK training of an infinitely wide network). Then $$\kappa=\lambda+\sum_\rho\frac{\kappa\eta_\rho}{\kappa+P\eta_\rho},\qquad \gamma=\sum_\rho\frac{P\eta_\rho^2}{(\kappa+P\eta_\rho)^2},\qquad E_g=\frac{1}{1-\gamma}\sum_\rho\frac{\eta_\rho\left(\kappa^2\bar w_\rho^2+\sigma^2P\eta_\rho\right)}{(\kappa+P\eta_\rho)^2}.$$ Each mode is fit like ridge on its own eigenvalue with penalty $\kappa/P$, and $\kappa$ is $\lambda$ plus the kernel mass of the modes the data cannot yet resolve; $\gamma$ is the variance factor $\mathrm{df}_2/n$ of linear ridge.

The error is a quadratic form in which the teacher and the sample separate. With noiseless labels, $\bar w-w=\lambda(\Psi\Psi^\top+\lambda I)^{-1}\bar w$, and whitening gives $$E_g=\mathrm{Tr}\,\big(D\,\langle G^2\rangle\big),\qquad G=\Big(\tfrac1\lambda\Phi\Phi^\top+\Lambda^{-1}\Big)^{-1},\quad D=\Lambda^{-1/2}\bar w\bar w^\top\Lambda^{-1/2},$$ with the target only in $D$ and the sample only in $G$, whose whitened columns have identity second moment. The rest is computing $\langle G^2\rangle$.

Adding one sample gives a PDE in the sample size. Reach the second moment through a first moment by shifting, $\tilde G(p,v)=(\frac1\lambda\Phi\Phi^\top+\Lambda^{-1}+vI)^{-1}$, so $G^2=-\partial_v\tilde G$ at $v=0$. A new sample is a rank-one update, and Sollich's approximation averages the Sherman-Morrison numerator and denominator separately; with $p$ continuous, $$\partial_p\langle\tilde G\rangle=\frac{1}{\lambda+\mathrm{Tr}\langle\tilde G\rangle}\,\partial_v\langle\tilde G\rangle,\qquad \langle\tilde G(0,v)\rangle=(\Lambda^{-1}+vI)^{-1}.$$

Characteristics give the self-consistent ridge. The trace $t=\mathrm{Tr}\langle\tilde G\rangle$ solves $\partial_pt=(\lambda+t)^{-1}\partial_vt$, so it is constant along lines on which $v$ falls at rate $1/(\lambda+t)$; following one back to $p=0$, $$g_\rho(p,v)=\Big(\frac1{\eta_\rho}+v+\frac{p}{\lambda+t}\Big)^{-1},\qquad t(p,v)=\sum_\rho g_\rho(p,v).$$ At $v=0$, with $\kappa=\lambda+t$, the second identity is the $\kappa$ equation: each mode is ridge on its own eigenvalue with penalty $\kappa/p$.

Differentiating in $v$ gives the mode errors and the shared factor. $\langle G^2\rangle_{\rho\rho}=-\partial_vg_\rho|_{v=0}$, and the chain rule through $t(p,v)$ gives $$E_\rho=\eta_\rho\bar w_\rho^2\,\frac{\kappa^2}{(\kappa+P\eta_\rho)^2}\cdot\frac{1}{1-\gamma}.$$ The first factor is the mode's ridge shrinkage; the second, shared by all modes, accounts for fluctuations of the sample Gram matrix, and with noise it amplifies an added $\sigma^2\gamma/(1-\gamma)$.

Spectral bias and task alignment follow from the shape of $E_\rho$. Differentiating the $\kappa$ equation gives $\kappa'(P)=-\kappa\gamma/(P(1-\gamma))\lt0$, and then $$\frac{d}{dP}\log\frac{E_\rho}{E_{\rho'}}=-2\Big[\frac{\kappa'+\eta_\rho}{\kappa+P\eta_\rho}-\frac{\kappa'+\eta_{\rho'}}{\kappa+P\eta_{\rho'}}\Big]\lt0\quad\text{for }\eta_\rho\gt\eta_{\rho'},$$ with ratio 1 at $P=0$, so normalized errors are ordered opposite to the eigenvalues at every $P$. By summation by parts, a target whose cumulative power curve $C(\rho)$ is pointwise larger has lower $E_g(P)/E_g(0)$ at every $P$.

Power-law spectra give power-law curves. Ridgeless, the substitution $\rho=(P/\kappa)^{1/b}u$ in the $\kappa$ equation gives $\kappa\propto P^{1-b}$, so the cutoff $P\eta_{\rho^*}=\kappa$ sits at $\rho^*\propto P$, $\gamma$ tends to a constant, and the error splits there: $$E_g\asymp\sum_{\rho\gt P}\rho^{-a}+\sum_{\rho\lt P}\rho^{-a}\Big(\frac{\kappa}{P\eta_\rho}\Big)^2\asymp P^{-(a-1)}+P^{-2b}\sum_{\rho\lt P}\rho^{2b-a}.$$ The unlearned tail gives $P^{-(a-1)}$; the residual shrinkage of learned modes gives $P^{-2b}$ only for very smooth targets, $a-1\gt2b$, so the kernel caps the rate at $2b$. With a fixed ridge, once $P^{1-b}\lt\lambda$, $\kappa\approx\lambda$, the cutoff moves to $(P/\lambda)^{1/b}$, and both exponents are divided by $b$. A numerical solve reproduces the slopes ($-1.50$ for $b=2,a=2.5$; $-3.99$ for $b=2,a=6$).

On the sphere each polynomial degree is a white band-limited problem. A dot-product kernel on $S^{D-1}$ has degree-$k$ multiplicity $N(D,k)\sim D^k/k!$ and $\eta_k\sim N(D,k)^{-1}$. At $P=\alpha N(D,L)$, lower degrees have $P\eta_K\to\infty$ and drop out as learned; higher ones keep their power, add their kernel mass to $\kappa$ like a multiple of the identity, and enter the variance exactly as label noise does. What remains is $N(D,L)$ equal eigenvalues, where the $\kappa$ equation is quadratic: the noiseless ridgeless error is $(1-\alpha)_+$ times the stage power, with noise it peaks at $\alpha=1$ ($\gamma\to1$), and the optimal ridge equals the effective noise at every $\alpha$.

The averages reduce to traces of resolvents. Expanding in $\epsilon$ splits the loss into a label term and a noise term, and the $w$ average is one Gaussian contraction. Two identities are used, $q^2=-\partial_\gamma q$ and $(\gamma I+\phi\phi^\top)q=I$ for $q=(\gamma I_N+\phi\phi^\top)^{-1}$, and the test average replaces $\hat\phi\hat\phi^\top$ by $\hat T\,\Omega$, $\Omega=u\Lambda u^\top$, so $$\text{noise term}=\frac{\sigma_\epsilon^2}{2}\Big\langle\Big(1+\gamma\frac{\partial}{\partial\gamma}\Big)\mathrm{tr}\{\Omega\bar q\}\Big\rangle_u,\qquad \bar q=\langle q\rangle_x ,$$ and what remains is the training-set average of one resolvent.

That average is the Marchenko-Pastur self-consistency, found by summing planar diagrams. For fixed $u$ the columns of $\phi$ are i.i.d. $N(0,\Omega)$; expanding $q$ in $1/\gamma$ and keeping the non-crossing Wick pairings, which dominate at large $N,T$, resums to $$\bar q=\Big(\gamma I_N+\frac{T\,\Omega}{1+\Delta}\Big)^{-1},\qquad \Delta=\mathrm{tr}\{\Omega\bar q\}.$$ As $\gamma\to0$, $\Delta=\Delta_{-1}/\gamma+\Delta_0+O(\gamma)$, and $1+\gamma\partial_\gamma$ removes the pole, so the ridgeless noise term is $\frac{\sigma_\epsilon^2}2\Delta_0$, which underparameterized is the least-squares variance $N/(T-N)$ whatever the spectrum.

The label term is an unseen variance times a variance inflation factor. The same machinery leaves a latent-space resolvent whose $u$ average has the same form as the $x$ average, and the prefactor drops out: $$L=\frac{\sigma_w^2}{2M}\cdot\frac{\Delta_{-1}(N)}{1-N/T}\qquad(N\lt T).$$ The numerator is the latent covariance projected onto the null space of the $N$ random directions, the variance the features cannot see. The factor $1/(1-N/T)=1+\Delta_0$ is least-squares variance inflation: the random features are noisy proxies of the latent ones, so even noiseless labels look noisy, which is why the loss peaks at $N=T$ with $\sigma_\epsilon=0$.

Data and features are dual. Since $\phi=ux$ with $x=\Lambda^{1/2}z$, the feature matrix is $G_1\Lambda^{1/2}G_2$ for i.i.d. Gaussian factors (our gloss), and its transpose has the same law with $N$ and $T$ exchanged. The paper rewrites the label term with the data-data resolvent and checks that swapping $x^\top$ with $u$ and $T$ with $N$ maps the $N\gt T$ computation onto the $N\lt T$ one, giving $\Delta_{-1}(T)/(1-T/N)$; the noise term is not symmetric, because the noise is added per sample.

The power-law solution, and why every exponent here is $\alpha$. Try $\Delta_{-1}=\lambda_+k\mu^{-\alpha}$ in the integral form of the defining equation; the substitution $I=s\,\mu\,k^{-1/(1+\alpha)}$ turns it into a Beta integral, $$k^{1/(1+\alpha)}=\int_0^\infty\frac{ds}{1+s^{1+\alpha}}.$$ The defining equation is the ridgeless $\kappa$ equation with $\eta_I=\lambda_I$ and $P=\mu$, so $\Delta_{-1}=\kappa$. An isotropic teacher puts power $\sigma_w^2\lambda_I/M$ on mode $I$; since the $\kappa$ equation gives $\kappa\sum_I\lambda_I(\kappa+\mu\lambda_I)^{-2}=1-\gamma$, Canatar's error is then exactly $\sigma_w^2\kappa/M$, twice Maloney's label term at $N=\infty$ because his loss carries a $\frac12$ (our check). In Bordelon's notation $a=b=1+\alpha$ and $\min\{a-1,2b\}=\alpha$: the unlearned tail always dominates.

Consequences and limits. Two spectra fix the curve, and $\kappa$ turns a resource into a cutoff index: the exponent is a spectral decay rate, the plateau is the tail beyond the smaller resource, and the latent dimension ends the law. With label noise and a ridgeless power law, $\gamma\to1-1/b$ and the excess error tends to $\sigma^2(b-1)$, the tempered regime of Mallinar et al. (our computation from the formula, not in the papers). Limits: the derivations are non-rigorous (separate averaging, Gaussian features, replica symmetry, planar diagrams without error bounds), features are fixed (no feature learning), and training is to convergence, so compute does not appear.