bubbleforce: the edge of stability for Muon

On a fixed quadratic, each momentum kernel has a threshold set by the roots of a polynomial. Take $L(W)=\tfrac12\lambda\|W\|^2$ and write $\kappa=\eta\lambda$. The update $W_{t+1}=W_t-\kappa\sum_ic_iW_{t-i}$ is a linear recurrence whose solutions are sums of modes $W_t\propto z^t$, with $z$ a root of the characteristic polynomial $P_\kappa$the polynomial whose roots are the per-update growth factors of the recurrence's modes; a complex root $z$ gives a mode that scales by $|z|$ and rotates by the angle of $z$ each update,$$P_\kappa(z)=z^{K+1}-z^K+\kappa\sum_{i=0}^{K}c_iz^{K-i}.$$A root with $|z|>1$ is a mode that grows at every update, so the recurrence is stable when the root radius $\rho(P_\kappa)$the largest modulus of a root, $\rho(P)=\max\{|z|:P(z)=0\}$ is below 1. Small $\kappa>0$ is stable, and bubbleforce is the first $\kappa$ at which a root reaches the unit circle,$$\kappa_\star(c)=\min\{\kappa>0:\rho(P_\kappa)=1\}.$$For gradient descent $P_\kappa(z)=z-1+\kappa$, whose root $1-\kappa$ leaves the circle at $-1$ when $\kappa=2$. For untruncated heavy-ball plus Nesterov momentum the moving average is one extra state, and $P_\kappa(z)=z^2-(1+\beta-(1-\nu\beta)\kappa)z+\beta(1-(1-\nu)\kappa)$. A nonreal pair of roots has squared modulus equal to the constant term, at most $\beta<1$, so it never reaches the circle; $P_\kappa(1)=\kappa(1-\beta)>0$ rules out $+1$; the exit is at $z=-1$, a period-two oscillation, and $P_\kappa(-1)=0$ gives Gitman et al.'s (2019) threshold$$\kappa_\star=\frac{2(1+\beta)}{1+\beta-2\nu\beta}.$$For a Hessian $H$the Hessian $\nabla^2L$ of the loss, constant on a quadratic, with eigenvalues $\lambda_1,\ldots,\lambda_d$ the eigendirections decouple with $\kappa_j=\eta\lambda_j$, which gives the theorem above. At $\beta=\nu=0.95$, $\kappa_\star\approx26.9$ (my check, unchanged by the 256-weight truncation), about 13 times gradient descent's 2. Lee and Lee (2026) give this general construction concurrently; the post's own contribution is the extension to Muon.

For gradient descent, oscillation pushes the sharpness down, so training settles at the threshold. The argument is Damian et al.'s (2023). Write $W_t=\bar W+x_tu$, with $\bar W$the local time average of the iterates, the center the run oscillates around the center of the oscillation, $u$the unit top eigenvector of the Hessian at $\bar W$ the top Hessian eigenvector and $x_t$the signed displacement of $W_t$ from $\bar W$ along $u$ the displacement along it. Averaging the gradient over a symmetric oscillation removes the term linear in $x_t$, and for a simple top eigenvalue the quadratic term, which comes from the loss's third derivative, is the gradient of the sharpness:$$\begin{aligned}\mathbb E[\nabla L(W_t)]&\approx\nabla L(\bar W)\\&\quad+\underbrace{\tfrac12\,\mathbb E[x_t^2]\,\nabla\lambda(\bar W)}_{\text{sharpness penalty}}.\end{aligned}$$The oscillation thus performs gradient descent on the sharpness until $\eta\lambda<2$ and it dies, and progressive sharpeningthe steady rise of the sharpness during training while it is below the threshold; it is observed, and has no accepted explanation pushes $\lambda$ back up. The post's Conjecture 1 extends this to any kernel, $\eta\lambda_t\to\kappa_\star$, under persistent progressive sharpeningwhy: something must keep raising the curvature; without it the run can sit below the threshold and nothing brings it to the edge and a slowly moving Hessianwhy: the threshold of Step 1 is computed for a fixed $H$; it applies to a real run only if $H$ changes little over the updates the kernel remembers; the second fails in practice, since top Hessian eigenvectors turn by about 50° per update.

For Muon the pathspring takes the place of the sharpness, and the same polynomial applies. Freeze the past gradients at update $t$; the update is then the optimizer operatorthe update as a function of the current weights with past gradients frozen $\mathcal T(W)=A\big(c_0\nabla L(W)+\sum_{i>0}c_i\nabla L(W_{t-i})\big)$, with $W_{t+1}=W_t-\eta\mathcal T(W_t)$. By the chain rule its Jacobian at $W_t$ is$$J_{\mathcal T}(W_t)=c_0\,J_{\mathrm{msign}}(M_t)\,H(W_t),$$with $J_{\mathrm{msign}}(M_t)$the derivative of the polar factor $\mathrm{msign}$ at the momentum $M_t$, a linear map on matrices and $H(W_t)$ the Hessian. The pathspring is $\zeta_t=\lambda^{+}_{\max}\big(J_{\mathcal T}(W_t)/c_0\big)$, the largest positive real eigenvalue of $J_{\mathrm{msign}}(M_t)H(W_t)$, defined for $c_0>0$why: the definition divides by $c_0$; a kernel with no weight on the current gradient makes $\mathcal T$ independent of the current weights. A displacement $\delta W$ of the weights changes the momentum by $\delta M_t\approx\sum_ic_iH\,\delta W_{t-i}$ and the update by $J_{\mathrm{msign}}(M_t)[\delta M_t]$. Holding $H$ and $J_{\mathrm{msign}}$ fixed over the kernel's window, a displacement along an eigenvector of $J_{\mathrm{msign}}H$ with eigenvalue $\zeta_t$ obeys$$\delta W_{t+1}=\delta W_t-\eta\zeta_t\sum_ic_i\,\delta W_{t-i},$$the recurrence of Step 1 with $\kappa=\eta\zeta_t$. So Muon's threshold is $\eta\zeta=\htmlData{gr=gn8}{\kappa_\star}$, with the same $\kappa_\star$ as SGD with that kernel, and the post's Conjecture 2 is that non-adaptive optimizers settle there; for SGD, $\zeta$ is the sharpness. The post measures $\zeta$ by Arnoldi iterationa Krylov method that finds a few extreme eigenvalues of a large, possibly nonsymmetric matrix using only matrix-vector products on $J_{\mathcal T}/c_0$, each product a Hessian-vector product$Hv$, computed by differentiating $\nabla L\cdot v$ once more, without forming $H$ followed by a Jacobian-vector product$J_{\mathrm{msign}}(M)[V]$, the first-order change in Muon's update when the momentum moves by $V$, computed by differentiating through the Newton–Schulz iterations that Muon uses to approximate the polar factor: 6 to 13 minutes per value on 8 H100 GPUs.

After a change of learning rate or momentum, the pathspring moves to the new predicted level. The baseline was run at constant $\eta=0.025$ through update 2,009 and then changed once: all learning rates halved or doubled (the AdamW ones too, per the code's configs), or $\beta=\nu$ lowered to 0.8991 or raised to 0.9751 to halve or double $\kappa_\star$. The measurements spike above the prediction, at times several-fold, and what the law matches is their floor, which the post estimates with a lower envelopea curve through the lowest level around which the measurements repeatedly cluster within a window of 400 updates on each side. In every case the floor moves from the shared prediction to the new one: the post says within a few updates; by my reading of the figures, about two updates after the momentum decrease and a few hundred, approaching from above, after the doubled learning rate. After the momentum increase the floor stays about 10% below the prediction for the 4,000 updates measured, and the mean sits on it; the authors suspect the kernel's doubled memory (mean gradient age 40 updates instead of 20).

The prediction tracks the pathspring through a decaying learning rate and a changing kernel. The post has no theory for a moving target. In the unmodified baseline the prediction rises as $\eta$ decays, and the measurements follow it, about fivefold, to near update 2,700; after that the prediction grows without bound and the measurements level off, because late in a linear decay $\eta$ falls by a large fraction per update, faster than $\zeta$ can settle (the authors' explanation). A harder test is the authors' MaxEntSlopthe authors' momentum kernel: the maximum-entropy weights over 256 lags under linear constraints on $c_0$, $c_1$, total weight, mean lag and spread, and odd against even lags kernel, maximum-entropy weights under linear constraints that include equal total weight on odd and even lags, $\sum_i(-1)^ic_i=0$; from update 1,000 its weights move linearly between two such solutions (the code's config). That constraint makes $P_\kappa(-1)=\pm2$ for every $\kappa$ (my check), which closes the period-two exit, so the roots that reach the circle first are a nonreal pair. The measured $\zeta$ again follows the prediction to about update 2,700, somewhat above it early on (my reading of the figure). MaxEntSlop's speedrun submission, which also changes the weight-decay schedule, takes 3,060 updates instead of 3,250 (190 fewer; the post also says 200); the authors do not recommend it.

Learning rate and momentum together can steer the pathspring along a chosen path. Choosing $\eta_t$ and $\beta_t$ so that $\kappa_\star(\beta_t)/\eta_t$ equals a target should make $\zeta$ follow it. With $\eta$ fixed at 0.0125 and the momentum scheduled, the target traced the upper outline of a Bulbasaur drawing over roughly updates 3,500 to 11,500 (read from the figure); most measurements stayed within 15% of it, and the validation loss kept falling. The lower outline, near a quarter of that level, failed: after about 3,000 updates $\zeta$ ran to 2 to 10 times the target, the losses stopped falling at about the same time, and a simplified outline and other learning rates also failed. The authors guess that progressive sharpening makes low targets unstable.