The neural tangent kernel

Gradient descent on the parameters is kernel gradient descent on the function. By the chain rule $\partial_tf_t=\sum_p\partial_{\theta_p}f\,\langle\partial_{\theta_p}f,d_t\rangle_{p^{in}}=\Phi_{\Theta(t)}\langle d_t,\cdot\rangle$, and $\partial_tC(f_t)=-\|d_t\|^2_{\Theta(t)}$, so a convex cost reaches its minimum while $\Theta$ stays positive definite. Since $\Theta$ depends on $\theta$, the argument needs two facts: the kernel becomes deterministic as width grows, and it does not move during training.

At initialization the kernel converges, by induction on depth. Add one random layer on top of a depth-$L$ network. Its own parameters contribute $\frac1{n_L}\alpha^{(L)}(x)^\top\alpha^{(L)}(x')+\beta^2\to\Sigma^{(L+1)}$ by the law of large numbers, as in the NNGP induction. The lower layers contribute, by the chain rule, $$\frac1{n_L}\sum_{i,i'}\Theta^{(L)}_{ii'}(x,x')\,\dot\sigma(\tilde\alpha^{(L)}_i(x))\,\dot\sigma(\tilde\alpha^{(L)}_{i'}(x'))\,W^{(L)}_{ik}W^{(L)}_{i'k'},$$ and the induction hypothesis $\Theta^{(L)}_{ii'}\to\Theta^{(L)}_\infty\delta_{ii'}$ (gradients of distinct units become orthogonal) collapses the double sum to one average over i.i.d. Gaussian preactivations, which converges to $\Theta^{(L)}_\infty\dot\Sigma^{(L+1)}\delta_{kk'}$. Taking the widths to infinity one at a time makes each step a plain law of large numbers. So $\Sigma^{(L+1)}$ is the last layer's learning and $\Theta^{(L)}_\infty\dot\Sigma^{(L+1)}$ everything below it.

During training each weight and activation moves by $O(1/\sqrt n)$. Induct on depth again and close with Grönwall. The top-layer weights move at speed $n_L^{-1/2}\langle\alpha^{(L)}_i,d_{t}\rangle_{p^{in}}$, and the preactivations below move through the frozen sub-network kernel applied to a back-propagated direction that carries another factor $n_L^{-1/2}W^{(L)}$. By Cauchy-Schwarz both speeds are at most $n_L^{-1/2}\|d_t\|$ times the current sizes, so $A(t)$, the initial norms plus displacements, obeys $$A(t)\le A(0)\exp\Big(\frac{\max(c^2\|\Theta^{(L)}_\infty\|_{op},1)}{\sqrt{n_L}}\int_0^t\|d_s\|_{p^{in}}ds\Big),$$ with $c$ the Lipschitz constant of $\sigma$. The exponent vanishes, so every displacement is $O(n_L^{-1/2})$. The induction also needs the sub-network's driving direction to have bounded integral, which an a priori lemma gives: $n_\ell^{-1/2}(W^{(\ell)}(t)-W^{(\ell)}(0))\to0$ in operator norm for every layer.

So the kernel is constant, while the function still moves. Each kernel entry is an average over units of products of activations, $\dot\sigma$ of preactivations, weights and the sub-network kernel, each moving by $O(n_L^{-1/2})$ (for $\dot\sigma$ this is where bounded $\ddot\sigma$ enters), so the kernel moves by $O(n_L^{-1/2})$ uniformly on $[0,T]$. This is the lazy mechanism in its first form: no feature changes at infinite width (the preactivations stay Gaussian with covariance $\Sigma^{(\ell)}$), yet the output moves by order one because $n$ units each contributing $O(1/n)$ act coherently.

The limit kernel is positive definite on the sphere, through Hermite coefficients. The first term of the recursion is a Schur product of positive semidefinite kernels, so it suffices that $\Sigma^{(L+1)}$ is strictly positive definite, which propagates up the layers because a nondegenerate Gaussian vector and a nonconstant $\sigma$ force $\mathbb E(\sum_ic_i\sigma(g(x_i)))^2+\beta^2(\sum_ic_i)^2>0$ for $c\ne0$. The base case is a dot-product kernel; expanding a rescaled $\sigma$ in Hermite polynomials, $\mu=\sum_ia_ih_i$, gives $\mathbb E\mu(X)\mu(Y)=\sum_ia_i^2\rho^i$ for unit Gaussians with correlation $\rho$, so $$\Sigma^{(2)}(x,x')=\nu(x^\top x'),\qquad \nu(\rho)=\beta^2+\sum_ia_i^2\Big(\frac{n_0\beta^2+\rho}{n_0\beta^2+1}\Big)^i.$$ A non-polynomial $\sigma$ has infinitely many $a_i\ne0$, and because $n_0\beta^2>0$ every power-series coefficient of $\nu$ is positive, so Schoenberg's criterion in Gneiting's form gives strict positive definiteness. The bias is needed: with $\beta=0$ and an odd $\sigma$ the network is odd in $x$ and the kernel is singular on every pair $\{x,-x\}$ (our remark, not in the paper, which assumes $\beta>0$ throughout).

Least squares is a linear ODE. With $d_t=f^*-f_t$ and a constant kernel, $\partial_tf_t=\Pi(f^*-f_t)$ is linear and the matrix exponential gives $f_t$ and $f_\infty$ above. The top kernel principal components fit first, which the paper offers as motivation for early stopping. The mean of $f_\infty$ is the posterior mean of a GP with prior $N(0,\Theta_\infty)$ given noiseless labels, but the network's prior is $N(0,\Sigma^{(L)})$, so the second term's covariance is in general the posterior covariance of neither GP (our remark, not in the paper).

Laziness is set by scale, and width enters only through the normalization. One gradient step changes the objective, relatively, by about $\eta\|\nabla F\|^2/F$ and the Jacobian by at most $\eta\|\nabla F\|\,\|D^2h\|/\|Dh\|$; for the square loss the first dominates when $\kappa_h\ll1$ (a sufficient condition). Rescaling gives $\kappa_{\alpha h}=\alpha^{-1}\|\alpha h(w_0)-y^*\|\,\|D^2h\|/\|Dh\|^2$, which falls like $1/\alpha$ only if $h(w_0)=0$, arranged by a symmetrized initialization or by subtracting the initial function. A large $\alpha$ arises as an output multiplier; as the initialization scale $\lambda$ of a $q$-homogeneous model, which acts as $\lambda^q\alpha$; or as the normalization of $h_m=\alpha(m)\sum_{i\le m}\phi(\theta_i)$ with i.i.d. $\theta_i$ and $\mathbb E\phi(\theta_i)=0$. There $\|Dh\|^2\sim m\alpha(m)^2$, $\|h(w_0)\|\sim\sqrt m\,\alpha(m)$ by cancellation, and the Hessian is block diagonal across neurons, so $\|D^2h\|\le\alpha(m)\mathrm{Lip}(D\phi)$ does not grow with $m$. Hence $\mathbb E\kappa\lesssim m^{-1/2}+(m\alpha(m))^{-1}$: $\alpha(m)=1/\sqrt m$ is lazy as $m\to\infty$, and the mean-field $\alpha(m)=1/m$ is not.

On a finite horizon the comparison is Grönwall in function space. The parameters cannot travel far: by Cauchy-Schwarz and the energy identity $\frac{d}{dt}F_\alpha=-\|\nabla F_\alpha\|^2$, $$\int_0^T\|w_\alpha'\|\,dt\le\sqrt T\Big(\int_0^T\|\nabla F_\alpha\|^2\Big)^{1/2}\le\sqrt{T\,F_\alpha(w_0)}=\sqrt{T\,R(0)}/\alpha.$$ So $\Sigma(w_\alpha(t))-\Sigma(w_0)=O(1/\alpha)$ on $[0,T]$, and $\Delta=\|y-\bar y\|$ satisfies $\Delta'\le C_1/\alpha+C_2\Delta$, giving $\Delta=O(e^{C_2T}/\alpha)$. The parameter gap is one order smaller because the parameter velocity carries an extra $1/\alpha$.

For the square loss the exponential disappears, because the linear part of the gap has a sign. The residual is monotone, so the parameter speed is at most $\alpha^{-1}\|y(0)-y^*\|\mathrm{Lip}(h)$ and, under the condition on $\alpha$, the path stays in the ball of radius $r$ up to $T$. In $$\tfrac12\tfrac{d}{dt}\Delta^2=-\langle\Sigma(w_0)(y-\bar y),y-\bar y\rangle-\langle(\Sigma(w)-\Sigma(w_0))(y-y^*),y-\bar y\rangle$$ the first term is nonpositive and can be dropped, which is exactly what fails for a general loss. The forcing term grows linearly in $t$, and integrating to $T=K/\mathrm{Lip}(h)^2$ gives $K^2$ in place of an exponential.

Over-parameterized, the metric stays nondegenerate forever. The kernel is bounded below near $w_0$, and the flow converges fast enough never to leave. Within distance $r_0=\sigma_{\min}/(2\mathrm{Lip}(Dh))$ of $w_0$, $\Sigma(w)\succeq\sigma_{\min}^2/4$, and a strongly convex loss under a time-dependent metric bounded below by $\lambda$ converges linearly by Polyak-Łojasiewicz plus Grönwall. Integrating the parameter speed against this exponential bounds the total displacement, $$\int_0^\infty\|w_\alpha'\|\,dt\le\frac{8\kappa^{3/2}\|Dh(w_0)\|\,\|y(0)-y^*\|}{\alpha\,\sigma_{\min}^2},$$ which the assumptions keep below $r_0$, so the exit time is infinite and convergence is global. A stability lemma for the gap, forced by $(\Sigma(t)-\Sigma_0)\nabla R=O(1/\alpha)$, makes the comparison uniform in time.

Under-parameterized, straighten the image and get a local minimum. The constant-rank theorem flattens the image of $h$ near $h(w_0)$, an $r$-dimensional manifold, through local diffeomorphisms $\varphi$ of parameter space and $\psi$ of $\mathcal F$: $$\psi\circ h\circ\varphi^{-1}=\pi_r,$$ with $\pi_r$ keeping the first $r$ coordinates. In these coordinates $G_\alpha(g)=\alpha^{-2}R(\alpha\psi^{-1}(g))$ is strongly convex on a small chart for large $\alpha$ (its curvature correction is of order $1/\alpha+\|g\|$), with a minimizer at distance $O(1/\alpha)$, so the flow converges geometrically and the parameters never leave the chart. The limit is the best point of the image manifold near $h(w_0)$, which need not contain the minimizer of $R$.

The networks that work are not lazy. Two-layer ReLU students learn a 3-neuron teacher from $N(0,\tau^2)$ initialization, where 2-homogeneity makes $\tau^2$ act as $\alpha$. In $d=2$ small $\tau$ sends the neurons to the teacher directions, while $\tau=2$ interpolates with a tiny displacement and finds none; in $d=100$ test loss rises with $\tau$ to the linearized plateau, and so does the best test loss along training, so early stopping does not help. Widened and fully linearized, VGG-11 and ResNet-18 reach 61.7% and 56.7% on CIFAR-10, against 89.7% and 91.0% trained normally. The comparison has a confound: the linearized models also fit their training sets poorly (61% and 55% widened, well short of 100% along the $\alpha$ sweep), which for the sweep the authors trace to an ill-conditioned tangent kernel, whose top 0.2% of eigenvalues sit an order of magnitude above the rest, which decay like $1/i$. So the gap mixes a worse function class with harder optimization.

Consequences and limits. Any differentiable model can be made lazy by a choice of scale, and then training is guaranteed and fast: an infinitely wide NTK-parametrized network reaches a global minimum of any convex loss on the data whenever $\Theta_\infty$ is positive definite, at rates read off its spectrum and generalizes as kernel regression with an explicit kernel. The limitation comes from the same mechanism: features do not move, so the theory cannot account for feature learning, and a lazy model is a kernel method, the wrong one on hard tasks (compare the staircase paper of the high-dimensional SGD section). The scale is usually set implicitly, by the output normalization, the initialization variance or the $1/\sqrt m$ factor; deep networks with i.i.d. weights essentially need variance of order $1/\text{fan-in}$, so wide standard networks do become lazy, and Yang and Hu identify the output multiplier and learning-rate exponents as what moves a wide network out of this regime.