Convergence theory for Muon

Context. Muon orthogonalizes the momentum of each hidden matrix and, at scale, adds decoupled weight decay, $W\leftarrow(1-\eta\lambda)W-\eta\,UV^\top$, which Moonshot found essential for long runs. Kovalev (2025) reads Muon as a trust-region method and gives its first rates; Chen et al. (2025) show that with weight decay it solves a spectral-norm-constrained problem. In both, Muon's learning rate is a radius, so the size of the gradient never enters the step. A fixed-radius step moves the full distance $\eta$ even when its direction is off, so an error in the direction raises the loss by an amount proportional to $\eta$, the same order as the progress, and momentum is needed to remove the error floor that a single noisy gradient leaves. Decoupled weight decay plus a bounded update confines the weights to a spectral ball of radius $b/\lambda$ for an update of spectral norm at most $b$, the learning rate setting only how fast it is reached.

Muon is the exact trust-region step in the spectral norm. Minimizing the linear model over a spectral ball of radius $\eta$ gives $-\eta UV^\top$, because $\langle G,UV^\top\rangle=\|G\|_{\mathrm{nuc}}$ and $\|UV^\top\|_{\mathrm{op}}=1$, so Kovalev's method is $$m_{k+1}=(1-\alpha)m_k+\alpha\,g(x_k;\xi_k),\qquad x_{k+1}=x_k-\eta\,\mathrm{orth}(m_{k+1}).$$ A squared-norm penalty gives the same direction with length $\theta\|G\|_{\mathrm{nuc}}$; only the trust-region form makes a constant learning rate Muon, and it caps every step at $\eta$, which is what controls the drift of the momentum. The analysis assumes EMA momentum and exact orthogonalization, so the speedrun's Nesterov momentum and five Newton-Schulz steps are not covered.

A fixed-radius step lowers the loss by $\eta$ times the dual norm of the gradient. With exact gradients, the smoothness upper bound and the trust-region identity give $$f(x_{k+1})\le f(x_k)-\eta\|\nabla f(x_k)\|_*+\tfrac L2\eta^2,$$ and telescoping gives $\min_k\|\nabla f(x_k)\|_*\le\Delta_0/(\eta K)+L\eta/2$, so $\eta\asymp\varepsilon/L$ and $K\asymp L\Delta_0/\varepsilon^2$, the optimal deterministic rate. It is gradient descent's rate in the pair of norms: for Muon, $L$ is measured in the spectral norm and stationarity in the nuclear norm.

With momentum, the momentum error enters the descent inequality multiplied by $\eta$, like the progress. Feeding $m_{k+1}$ to the oracle and bounding the mismatch by the dual-norm pairing gives $$f(x_{k+1})\le f(x_k)-\eta\|\nabla f(x_{k+1})\|_*+2\eta\|\nabla f(x_{k+1})-m_{k+1}\|_*+\tfrac32L\eta^2 .$$ The error carries the same factor $\eta$ as the progress. In SGD the noise enters only at order $\eta^2$, because the cross term averages to $-\eta\|\nabla f\|^2$; here the step is a nonlinear function of its direction, so $\mathbb E\,g=\nabla f$ does not average the error away, and the direction itself must be close to $\nabla f$ in the dual norm.

The momentum error is initialization plus drift plus averaged noise. Unroll the EMA around the moving target. The initialization error decays like $(1-\alpha)^{k+1}$; each drift term $\nabla f(x_i)-\nabla f(x_{i+1})$ is at most $L\eta$ because the trust region caps every step, and the weights sum to at most $1/\alpha$; the noise terms are conditionally centered, so their sum is bounded in the Euclidean norm, where the cross terms vanish, and converted with $\rho=\sqrt{\min(m,n)}$, the factor from Frobenius to nuclear norm, with $\sigma^2$ the Euclidean gradient variance. Hence $$\mathbb E\|m_{k+1}-\nabla f(x_k)\|_*\le(1-\alpha)^{k+1}\rho\sigma+\sqrt\alpha\,\rho\sigma+\frac{L\eta}{\alpha}.$$

Momentum lowers the noise floor but adds staleness, and balancing the two gives the optimal rate. Inserting the error bound into the descent inequality, after an index shift that adds $2L\eta$, gives Kovalev's Theorem 2, $$\mathbb E\min_{k\le K}\|\nabla f(x_k)\|_*\le\frac{\Delta_0}{\eta K}+\frac{2\rho\sigma}{\alpha K}+2\sqrt\alpha\,\rho\sigma+\frac{7L\eta}{2}+\frac{2L\eta}{\alpha}.$$ Without momentum ($\alpha=1$) the bound keeps $2\rho\sigma$ for every $\eta$ and $K$, a floor only a larger batch lowers. Momentum averages about $1/\alpha$ gradients, which lowers the floor to $2\sqrt\alpha\,\rho\sigma$ and adds a staleness term $2L\eta/\alpha$. Balancing gives $K\asymp L\Delta_0\rho^2\sigma^2/\varepsilon^4$, the unimprovable stochastic first-order rate. This matches SGD's rate in order. It shows that Muon is no worse in the worst case; it does not show that Muon is faster, and any advantage would come from $L$, $\rho$ and the norm in which stationarity is measured. The same lemma argues, without a theorem, for momentum before orthogonalization: an average of raw gradients averages centered errors, whereas an average of $\mathrm{orth}(g_i)$ averages biased ones.

Weight decay shifts the ball toward the origin, which bounds the iterates and makes the analysis a contraction. Kovalev's weight-decay step is the trust region centered at $(1-\beta)x_k$, that is $x_{k+1}=(1-\beta)x_k-\eta\,\mathrm{orth}(m_{k+1})$. If $\eta\ge\beta\max\{\|x_0\|,\|x^*\|\}$, the iterates stay in $\|x\|\le\eta/\beta$ by induction, and the point $\beta x^*+(1-\beta)x_k$ is feasible; comparing the step with it, using smoothness and star-convexity of $f$, gives $$f(x_{k+1})-f^*\le(1-\beta)\big(f(x_k)-f^*\big)+4L\eta^2+2\eta\|m_{k+1}-\nabla f(x_k)\|_* .$$ Unrolling, with the momentum error bounded as before but with drift $2L\eta/\alpha$, since steps now reach $2\eta$, gives the contraction $(1-\beta)^K$. With $\eta=\beta D$, $D=\|x^*\|$, the noise floor becomes $2\sqrt\alpha\,D\rho\sigma$, set by $\alpha$ alone, and every other floor is linear in $\beta$, which gives $\varepsilon^{-3}$ in function value. In code, $p\leftarrow(1-\mathrm{lr}\cdot\mathrm{wd})\,p-\mathrm{lr}\cdot s\cdot\mathrm{orth}(m)$ with shape factor $s$ has $\beta=\mathrm{lr}\cdot\mathrm{wd}$ and $\eta=\mathrm{lr}\cdot s$, so the invariant ball $\|p\|_{\mathrm{op}}\le s/\mathrm{wd}$ does not depend on the learning rate.

Extrapolation cancels the first-order lag of momentum. The drift $L\eta/\alpha$ is the gap between an average of past gradients and the current one. Sampling the gradient at $\bar x_{k+1}=x_k+\frac1\alpha(x_{k+1}-x_k)$, chosen so that $\alpha\bar x_{k+1}+(1-\alpha)x_k=x_{k+1}$, makes each drift term a convex combination of gradients minus the gradient at the combination's own point, so the first-order Taylor term cancels, and a Hessian-Lipschitz condition with constant $H$ bounds the remainder by $$\frac H2\Big(\alpha\|\bar x_{i+1}-x_{i+1}\|^2+(1-\alpha)\|x_i-x_{i+1}\|^2\Big)\le\frac{H\eta^2}{2\alpha}.$$ Re-balancing gives $\varepsilon^{-3.5}$, from a new Muon variant whose practical value is open.

Muon is Lion-K with the nuclear norm, so with weight decay it is a constrained method. Chen et al.'s template keeps a momentum with weight $\beta_2$, takes the update at a look-ahead with weight $\beta_1$, and applies $\nabla K$ for a convex $K$; with decoupled decay it minimizes $F+\frac1\lambda K^*(\lambda X)$. Muon's $\mathrm{msgn}(X)=U\,\mathrm{sgn}(\Sigma)V^\top$ is a subgradient of the nuclear norm (the element of Watson's subdifferential with no extra term), and the conjugate of a norm is the indicator of the dual unit ball, here the spectral unit ball, so Muon with weight decay solves $$\min_X F(X)\quad\text{s.t.}\quad\|X\|_{\mathrm{op}}\le\frac1\lambda .$$ At a fixed point $M=\tilde M=-\nabla F(X)$, so $\lambda X=\mathrm{msgn}(-\nabla F(X))$; since $\partial K^*$ inverts $\partial K$, this is stationarity of the regularized objective, and every singular value on the gradient's range sits at $1/\lambda$, on the boundary. Practical Newton-Schulz is the same direction up to bounded distortion of the singular values (for a $256\times1024$ Gaussian $G$, $\langle G,\mathcal N(G)\rangle\approx0.95\,\|G\|_{\mathrm{nuc}}$, our computation), and the constraint applies to it unchanged.

Any bounded update with decoupled weight decay reaches a ball of radius $b/\lambda$ and stays there. The constraint does not depend on momentum or on the choice of $K$. For $X_{t+1}=X_t+\eta_t(O_t-\lambda X_t)$ with $\|O_t\|\le b$ in any norm and $\eta_t\lambda\le1$, the triangle inequality gives $$\|X_{t+1}\|-\frac b\lambda\le(1-\eta_t\lambda)\Big(\|X_t\|-\frac b\lambda\Big),$$ so the excess over $b/\lambda$ shrinks by $e^{-\lambda\sum_s\eta_s}$, and weights inside the ball stay inside. For Muon the norm is spectral and $b$ is the update's spectral norm as applied: the shape factor times the polynomial's overshoot (at most about 1.20 for Jordan's five steps and 1.12 for Polar Express, our computation). With exact orthogonalization $b=s$, and this is Kovalev's ball $s/\mathrm{wd}$ again.

What this means for choosing $\lambda$. Choosing $\lambda$ chooses a ball, of radius $b/\lambda$ in units of the update's spectral norm. With decay coupled to the learning rate as $1-\eta\lambda$, the learning rate does not enter the radius; it sets only the speed $\lambda\sum_s\eta_s$ at which the weights get there, which an LR cooldown slows. With SGD this decay is a ridge penalty; here it is a hard constraint, and the speedrun's cautious weight decay, which decays only where the update agrees in sign and keeps the original objective, is a different object. The experiments (ResNets, a ViT, small LLaMA and Qwen models) show top singular values staying below $1/\lambda$, which any bounded update with decay guarantees; whether the iterates reach KKT points, whether the constraint helps generalization, and whether it binds at the weight decay values used at scale are not tested.

The KKT score is $\lambda$ times the Frank-Wolfe gap. Chen et al. measure convergence by $S(X)=\|\nabla F\|_{\mathrm{nuc}}+\langle\lambda X,\nabla F\rangle$. For feasible $X$, $$\max_{\|Z\|_{\mathrm{op}}\le1/\lambda}\langle\nabla F(X),X-Z\rangle=\langle\nabla F,X\rangle+\tfrac1\lambda\|\nabla F\|_{\mathrm{nuc}}=\frac{S(X)}\lambda$$ (our reading), so $S\ge0$ by norm duality and $S=0$ means no feasible direction decreases the linear model. Their Proposition 2 makes this exact: $S(X)=0$ puts $\lambda X$ in the subdifferential of the nuclear norm at $-\nabla F$, so the singular values of $X$ on the gradient's range sit at $1/\lambda$ with the singular vectors of $-\nabla F$, and feasibility with $S=0$ is the KKT condition.

Inside the ball a Hamiltonian-type Lyapunov function decreases, which needs $\beta_1\lt\beta_2$. With momentum written as $M\approx-\nabla F$, the Lyapunov function is $$H_t=F(X_t)-F^\star+\frac{c_t}{\lambda}\Big(\|M_t\|_{\mathrm{nuc}}-\langle\lambda X_t,M_t\rangle\Big),$$ whose second term is a kinetic energy, nonnegative by duality and zero when the weights are aligned with the momentum's polar factor at the boundary. Expanding $H_{t+1}-H_t$ with smoothness and convexity and choosing the weights so the cross terms cancel leaves a decrease by inner products that are nonnegative because subgradients of convex functions are monotone, plus $O(\eta^2C_K^2L)$ with $C_K=\sqrt{\min(n,m)}$. The weights divide by $\beta_2-\beta_1$, so plain EMA momentum ($\beta_1=\beta_2$) is excluded. The speedrun's Nesterov form, $m\leftarrow\beta m+(1-\beta)g$ with direction $\beta m+(1-\beta)g=\beta^2m_{\mathrm{old}}+(1-\beta^2)g$, has $\beta_2=\beta$ and $\beta_1=\beta^2\lt\beta_2$ (our mapping), so it is covered.

From the Lyapunov decrease to the KKT score, with a noise floor that momentum does not reduce. Write $S(X_t)=\langle\mathrm{msgn}(\nabla F)+\lambda X_t,\nabla F\rangle$, split $\nabla F$ into $-\tilde M_t$ plus the tracking error, and drop one term by norm duality: $$S(X_t)\le\Gamma_{t-1}+2C_K\|\nabla F(X_t)+\tilde M_t\|_F,$$ where $\Gamma_{t-1}\ge0$ is a monotonicity term of the Lyapunov decrease and the tracking error is of order $\eta$ plus a decaying initial term, by unrolling the EMA, since each step moves $X$ by at most $2\eta C_K$. With exact $\mathrm{msgn}$ and $\eta\asymp1/\sqrt T$, $\min_tS(X_t)=O(1/\sqrt T)$. With stochastic gradients (Chen et al.'s Theorem 4) a term $O(\sigma/\sqrt{n_{\mathrm{batch}}})$ remains that momentum does not reduce, because the noise is bounded step by step, never averaged; in Kovalev's analysis, which tunes the EMA weight to the target accuracy, momentum does remove it. An almost-sure version comes from a stochastic LaSalle argument with Robbins-Monro steps and growing batches.