Benign, Tempered, or Catastrophic

Context. Benign-overfitting theory reaches Bayes risk only with dimension growing with $n$ or a very slowly decaying spectrum. Networks trained to interpolation on fixed-dimension data behave differently: their test error neither diverges nor reaches Bayes risk as $n\to\infty$, and it grows with the label noise. The paper names this middle regime tempered and uses kernel regression to show which spectral property selects each regime: eigendecay slower than every power law is benign, a power law is tempered, and faster than every power law is catastrophic. On these notes' reading of the formula (not the paper's wording), the learnabilities of ridgeless kernel regression sum to exactly $n$ over the eigenmodes, and noise is amplified by the inverse of the share of that total on partially learned modes. For $\lambda_i=i^{-\alpha}$ that share tends to $1/\alpha$, so the risk tends to $\alpha\sigma^2$.

Setting: the taxonomy. A procedure with expected risk $R_n$ is benign if $\lim R_n=R^*$, the Bayes risk, tempered if $\lim R_n\in(R^*,\infty)$, and catastrophic if $\lim R_n=\infty$ (random guessing, $1-1/K$, for $K$-class classification). The diagnostic is the noise profile, the limiting clean test risk as a function of the training-label noise: flat at Bayes when benign, a finite increasing curve when tempered. Examples: $k$-NN with $k\sim\log n$ is benign, 1-NN tempered, random-features regression at $p=n$ catastrophic.

Setting: kernel regression and the risk estimate. Labels are $y_i=f^*(x_i)+\eta_i$, $\eta_i\sim N(0,\sigma^2)$, KR with ridge $\delta\ge0$ is $\hat f(x)=K(x,D_n)(K(D_n,D_n)+\delta I_n)^{-1}Y$ (with $\delta$, not $n\delta$), and $f^*=\sum_iv_i\phi_i$ in the Mercer eigenbasis $(\lambda_i,\phi_i)$. The paper analyzes the eigenlearning estimate of Simon et al. (equivalent to Bordelon et al. and others), cited without derivation, nonrigorous, expected to become exact as $n\to\infty$: $$\mathcal E_n=\mathcal E_0\Big(\sum_i(1-\mathcal L_i)^2v_i^2+\sigma^2\Big),\qquad \mathcal E_0=\frac{n}{n-\sum_j\mathcal L_j^2},\qquad \mathcal L_i=\frac{\lambda_i}{\lambda_i+\kappa},\qquad \sum_i\frac{\lambda_i}{\lambda_i+\kappa}+\frac{\delta}{\kappa}=n.$$ $\mathcal L_i\in[0,1]$ is the learnability of mode $i$, $\kappa$ the effective regularization, $\mathcal E_0\ge1$ the overfitting coefficient. Assumption A.1: eigenvalues strictly decreasing with finite trace, $\sum_iv_i^2<\infty$, and (e) the target weight on modes below $\lambda$ vanishes as $\lambda\to0$ (weight outside the RKHS counts as noise).

Theorem 3.1 (KR trichotomy). Under Assumption A.1 with $\sigma^2>0$: (a) if $\delta>0$, or if $\delta=0$ and $\lambda_i=i^{-1}\log^{-\alpha}i$ ($i\ge2$) for some $\alpha>1$, then $\lim_n\mathcal E_n=\sigma^2$ (benign); (b) if $\delta=0$ and $\lambda_i=i^{-\alpha}$, $\alpha>1$, then $\lim_n\mathcal E_n=\alpha\sigma^2$, so the excess risk is $(\alpha-1)\sigma^2$ (tempered); (c) if $\delta=0$ and $\lambda_i=i^{-\log i}$, or more generally $\dfrac{\lambda_i}{\lambda_{i+1}}\ge\dfrac{i^{-\log i}}{(i+1)^{-\log(i+1)}}$ for all $i$, then $\lim_n\mathcal E_n=\infty$ (catastrophic). Clause (a) with $\delta=0$ is the benign condition of Bartlett et al. (2020), and (b) says the power laws they showed are not benign are tempered.

Consequences for named kernels (heuristic). Only the spectral tail matters, so each named kernel is matched to a clause by its known tail (not a formal corollary: spherical-harmonic spectra are degenerate and only asymptotically of the stated form). The Laplace kernel on a $d$-manifold and the ReLU NTK on the sphere have $\lambda_i\sim i^{-(d+1)/d}$, so clause (b) with $\alpha=1+1/d$ would give excess $\sigma^2/d$; the paper claims only $\Theta(1/d)$, tempered and approaching benign as $d$ grows. The Gaussian kernel has $\lambda_i\sim e^{-ci^{2/d}}$, whose ratio eventually beats that of clause (c), so ridgeless Gaussian KR is catastrophic. Early stopping acts as a ridge, so early-stopped wide networks should be benign, and since nearly any kernel on the sphere is the NTK of some activation, NTK-regime networks can be tempered or catastrophic.

Empirical results. On pure-noise labels, ridge Gaussian KR goes to 0, ridgeless Laplace KR plateaus with excess about $1/d$, and ridgeless Gaussian KR diverges. Synthetic KR with exact spectra $i^{-\alpha}$ reaches $\approx\alpha\sigma^2$. Interpolating networks (ResNets, VGG, MLPs) have noise profiles converging to a nonzero, roughly linear curve; early-stopped versions approach benign.

Consequences and limits. Interpolation alone does not determine the regime; the regime is set by how sharply learnability falls off at index $n$, that is, by the spectral tail exponent. A heuristic consistent with all three clauses, not in the paper: $\mathcal E_0\approx$ the local exponent $-d\log\lambda/d\log i$ at $n$, which tends to 1, equals $\alpha$, and is $2\log n$ in the three cases (numerically $\mathcal E_0=13.8$ at $n=1000$ for $i^{-\log i}$), though rough when the exponent drifts. In fixed dimension, then, common kernels are tempered, smooth kernels catastrophic, and a ridge or early stopping benign, as with networks. Limits: the estimate is nonrigorous; the trichotomy is proved only for three exact spectral families; the network results are empirical, with the NTK link heuristic; and why classification noise profiles have slope near one, while KR gives $\alpha-1$, is unexplained.