Maximal update parametrization

One step of a one-hidden-layer network shows why the exponents matter. Take $f=Vx$, $x=\phi(U\xi)$, $V=n^{-a_v}v$, $U=n^{-a_u}u$, and $\chi_0=\mathcal L'(f_0,y_0)=\Theta(1)$. One step moves the preactivations by $$\Delta h_1(\xi)=-n^{-2a_u}\chi_0\xi_0\xi\,(V_0^\top\odot\phi'(h_0)).$$ In NTK, $a_u=0$ and $V_0$ has coordinates $\Theta(n^{-1/2})$, so $\Delta x_1=\Theta(n^{-1/2})$ and the feature kernel does not move. The output still moves by $\Theta(1)$, through $V_0\Delta x_1=-\chi_0\xi_0\xi\sum_\alpha(V_{0\alpha})^2\phi'(h_0(\xi)_\alpha)\phi'(h_{0\alpha})$, a sum of $n$ positive-mean terms of size $1/n$, and through $\Delta V_1x_0$, while $\Delta V_1\Delta x_1\to0$. In $\mu$P ($a_u=-\frac12$, $V_0$ of size $1/n$) it gives $\Delta h_1=\Theta(n)\cdot\Theta(1/n)=\Theta(1)$, so features move, and with nonlinear $\phi$ the step is not linear in $\eta$, which rules out a kernel.

Stability comes down to comparing CLT and LLN exponents. At initialization $W^lx^{l-1}$ has coordinates of size $n^{1/2-a_l-b_l}$ by the CLT, which fixes $a_l+b_l$. A gradient step adds the rank-one $\delta W^l=-\eta n^{-c-2a_l}dh\,x^\top$, and in the next forward pass $$\delta W^l\bar x=-\eta\,n^{1-c-2a_l}\Big(\frac{x^\top\bar x}{n}\Big)dh,$$ where $x^\top\bar x/n$ has a nonzero limit because the new activation is correlated with the old one: a law of large numbers, which gives a full factor $n$. With $dh$ of the readout's size, $n^{-(a_{L+1}+b_{L+1})}+n^{-(2a_{L+1}+c)}$, this is $\Theta(n^{-r_l})$ with $r_l=\min(a_{L+1}+b_{L+1},2a_{L+1}+c)+c-1+2a_l+\mathbb 1(l=1)$, the indicator accounting for the fixed input dimension, and $r=\min_lr_l$. So $r\ge0$ means the features do not blow up, the other two conditions bound the readout's update and its initial weights acting on the feature change, and imposing maximal updates everywhere leaves only $\mu$P.

SP is forced into the kernel regime, and $\mu$P is the minimal fix. In a two-hidden-layer linear SP network $f=V\bar h$, $\bar h=Wh$, $h=U\xi$, with $U_{\alpha\beta}\sim N(0,1)$ and $W,V\sim N(0,1/n)$, one step on $W$ gives $f_1=f_0-n^{-c}\chi\,(VV^\top)(h^\top h)=f_0-\Theta(n^{1-c})$, since $h^\top h=\Theta(n)$ by the LLN. So $c\ge1$; then $\bar h$ moves by $O(n^{-1/2})$ per coordinate, $r=\frac12$, and $f_1-f_0\to-\chi\|\xi\|^2$, kernel descent with the linear kernel. The blowup comes from the LLN: at initialization $W^{L+1}x^L$ is a sum of independent zero-mean terms, but after one update $x^L_1$ is correlated with $W^{L+1}_1$ and the sum grows like $n^{1-a_{L+1}-b_{L+1}}$. Dividing the logits by $\sqrt n$ ($a_{L+1}=\frac12$) with $c=0$ keeps $f$ bounded and lets $\bar h$ move by $\Theta(1)$. The first layer still moves only by $O(1/n)$, since its backward vector has coordinates of size $1/n$; $a_1=-\frac12$, $b_1=\frac12$ (same forward pass, gradient scaled up by $n$) fixes that, and hidden layers keep the standard $N(0,1/n)$ initialization.

Tensor Programs compute every limit by tracking a typical coordinate. Write training as a program and give each vector $h$ a random variable $Z^h$ for a typical coordinate: Gaussian for initial vectors, the function applied for Nonlin, an expectation for Moment. A product $Wx$ splits as $Z^{Wx}=\hat Z^{Wx}+\dot Z^{Wx}$: the first part is jointly Gaussian with $\mathrm{Cov}(\hat Z^{Wx},\hat Z^{Wy})=\sigma_W^2\mathbb EZ^xZ^y$, as if $W$ were independent of $x$; the second corrects for using both $W$ and $W^\top$, as backpropagation does, by a Gaussian integration-by-parts (Stein) term $$\dot Z^{Wx}=\sigma_W^2\sum_iZ^{y^i}\,\mathbb E\frac{\partial Z^x}{\partial\hat Z^{W^\top y^i}}$$ over the vectors $y^i$ that $x$ depends on through $W^\top y^i$. For one hidden layer in $\mu$P the limit is a closed recursion (Theorem 6.1), $$\mathring f_t(\xi)=\mathbb E\,Z^{nV_t}\phi(\xi Z^{U_t}),\quad Z^{nV_{t+1}}=Z^{nV_t}-\mathring\chi_t\phi(\xi_tZ^{U_t}),\quad Z^{U_{t+1}}=Z^{U_t}-\mathring\chi_t\xi_tZ^{nV_t}\phi'(\xi_tZ^{U_t}).$$ For linear $\phi$, $Z^{U_t}$ and $Z^{nV_t}$ stay linear combinations of the two initial Gaussians, so the limit is a width-$(d+d_o)$ network with diagonal initialization, which makes the Word2Vec and MAML limits computable; for nonlinear networks the nesting of $\phi'$ inside $\phi$ (and, in deeper networks, the $\dot Z$ terms) makes the cost grow exponentially with the number of steps.

The dichotomy, from the limit equations. If $r>0$, every scalar multiplying a feature update vanishes, features stay at initialization, and the logit update unrolls into kernel gradient descent with an NTK-type kernel, assembled from feature kernels at initialization and products of $\mathbb E\phi'\phi'$ terms over the layers updated at leading order. If $r=0$, $\partial_\eta^2\mathbb E(Z^{x^L_1(\xi_0)})^2\ne0$ at $\eta=0$ (the first derivative vanishes) and $\partial_\eta^3\mathring f_1(\xi)\ne0$, so a small learning rate moves the feature kernel and the first step is not linear in $\eta$. Corollary 3.10 follows from the exponents: $r=0$ and stability force $a_{L+1}+b_{L+1}\ge1$, so $f_0$, of size $n^{1/2-a_{L+1}-b_{L+1}}$, vanishes, and a feature-learning limit has no Gaussian-process prior. Kernel limits trivialize transfer because the body acts on the output only through $W^{L+1}_0$, which a new head resamples.

In experiments the stable learning rate stays fixed with width under $\mu$P, and the $\mu$P limit learns features. On CIFAR-10 the largest stable learning rate scales like $1/\text{width}$ in SP and stays constant in $\mu$P. On Omniglot 1-shot 5-way with first-order MAML, the $\mu$P limit of a one-hidden-layer linear network reaches 66.42% meta-test accuracy, against 47.60% and 47.82% for the GP and NTK limits of a ReLU network. On Word2Vec analogies the $\mu$P limit scores 43.31% after text8 and 56.45% after fil9, while NTK and GP limits score essentially 0, since frozen embeddings make the answer uniform over the vocabulary. These are linear one-hidden-layer models chosen so the limit is computable; that deep nonlinear $\mu$P limits model practice is only suggested.