Refinements to Muon

If Muon's updates match AdamW's update size, AdamW's tuning transfers. A rank-$r$ orthogonal factor $O=U_rV_r^\top$ has $\|O\|_F^2=r$ spread over $AB$ entries, so $$\mathrm{RMS}(O)=\sqrt{\frac{r}{AB}}=\frac1{\sqrt{\max(A,B)}}\quad\text{at full rank,}$$ and a wide $[H,4H]$ MLP matrix takes per-entry steps half those of an $H\times H$ attention matrix. Moonshot multiplies each update by $0.2\sqrt{\max(A,B)}$, so every matrix steps with RMS 0.2, where AdamW's update RMS sits in practice (0.2 to 0.4, below its nominal 1 because $\beta_1<\beta_2$ and $\epsilon>0$); targets of 0.2 and 0.4 tie in a sweep, and the shape rule beats one global multiplier. In practice, the learning rate and decay tuned for AdamW serve Muon unchanged, and the AdamW-trained embeddings, head and norms share one schedule with the Muon matrices.

On long runs Muon needs weight decay, because its steps never shrink. As gradients fade nothing slows the weights' growth, and Moonshot sees weight and layer-output RMS leave the range where bf16 is precise. Decoupled decay settles them where it balances the fixed step, inside a spectral ball of radius about $0.2\sqrt{\max(A,B)}/\lambda$ (the spectral-constraint note in this section). At 800M parameters and 100B tokens, Muon without decay leads early and then falls behind Muon with decay, which also ends below AdamW; this rests on one pair of runs.

With both fixes, Muon needs 44 to 49% of AdamW's compute over the tested range, untuned. Five compute-optimal dense models (399M to 1.5B parameters) are trained with AdamW at grid-searched hyperparameters, and Muon reuses exactly those. Solving the two fitted power laws for equal loss (my computation) gives a compute ratio of 0.44 at the smallest budget and 0.49 at the largest, so the quoted "about 52%" sits just past the tested range; the fits imply the ratio rises with compute, but their exponents (0.052 and 0.054) are printed to three decimals and a common 0.053 would give a constant 0.47, so the trend is uncertain. The comparison is conservative for Muon (AdamW tuned, Muon not) and rests on little data (five points per optimizer, no repeats, one data set).

Orthogonalization leaves neurons with unequal update sizes. In a 1.1B model, Muon's update has flat singular values but row norms (one row per output neuron) that vary widely, while AdamW's are much more uniform. The paper only measures this; an explanation not in the paper follows. With $M=U\Sigma V^\top$ and $\varphi$ the composed Newton-Schulz polynomial, $$\big\|e_i^\top\mathrm{NS5}(M)\big\|^2=\sum_k\varphi(\sigma_k)^2\,U_{ik}^2 .$$ An exact polar factor of a full-rank wide matrix has unit rows. For tall matrices ($m>n$, like the paper's headline MLP up-projection) the squared row norms are leverage scores summing to $n$, uneven even when exact. For any shape, five fixed-coefficient steps leave the small singular values of a nearly low-rank momentum well below 1, so each row's norm is that neuron's leverage in the top few directions, and in a small check it tracked the neuron's gradient scale. The effect is largest exactly when the update is dominated by a few directions.

What the row normalization saves. At 1.1B and 5.4B parameters (50B tokens), NorMuon reaches AdamW's final loss in 21.74% and 13.91% fewer steps, against 10.43% and 6.08% for Muon, at 2.9% more step time than AdamW. In modded-nanogpt at 124M and 350M, with Muon's tuned hyperparameters reused, Muon needs 6% and 15% more iterations to match it. These are single runs at one learning rate per scale, and the margin over Muon shrinks with size.

Cautious weight decay decays only where the step already shrinks the weight. Plain decay changes the problem (with Muon its fixed points are KKT points of $\min f$ subject to $\|X\|_{\rm op}\le1/\lambda$); cautious decay masks it, for any base update $u_t$, with no new hyperparameter: $$x_{t+1}=x_t-\eta_t\big(u_t+\lambda\,\mathbb 1\{u_t\odot x_t\ge0\}\odot x_t\big).$$ At the baseline's tuned $\lambda$ it lowers final loss for AdamW, Lion and Muon from 338M to 2B parameters; on OLMo-1B, AdamW goes from 2.65 to 2.56 and Muon from 2.51 to 2.42, while a random mask of the same sparsity is worse than no decay at all, so the gain comes from which coordinates are decayed.

The mask makes the loss a Lyapunov function, so the limit is unbiased. For the gradient flow with cautious decay, $\dot x=-\nabla f-\lambda\,\mathbb 1\{\nabla f\odot x\ge0\}\odot x$, differentiate $f$ along the path: $$\frac{d}{dt}f(x_t)=-\|\nabla f\|_2^2-\lambda\big\|(\nabla f\odot x)^+\big\|_1\le0 .$$ The mask turns the cross term into a positive part; with plain decay it is $-\lambda\langle\nabla f,x\rangle$, of either sign, so decay can raise the loss and the flow settles where $\nabla f=-\lambda x$. LaSalle then puts the limit set inside $\{\nabla f=0\}$, the stationary points of $f$ itself (the momentum methods need a kinetic term and a mask-dependent correction). On the stationary set the decay acts as a tie-breaker, sliding along flat directions toward smaller magnitudes until it reaches a locally Pareto point (a stationary point with no nearby stationary point that is smaller or equal in every $|x_i|$). The preference is mild: on the paper's circle of minimizers, $f=\big((x-3)^2+(y-3)^2-1\big)^2$ (my check), cautious decay started at $(4.2,4.2)$ stays at the large-norm point $(3.71,3.71)$, since every tangent move there raises one coordinate.

How the speedrun used each (records #41 to #50). The record-holder's essay (Varun Srivastava, who set #38, #42, #43 and #46) and the READMEs give the sequence at 124M on 8xH100. Record #41 swapped in NorMuon with hyperparameters unchanged and removed 15 steps (0.7 s); #42 found Muon's learning rate about twice too high, halved it and reshaped the attention weights so the normalization runs along the output dimension, for 30 steps and 3.3 s, about 2% for the pair by the essay's count. Record #43 replaced plain decay, which stabilized training but raised the final loss at this training length, with cautious decay on the NorMuon matrices, its strength tied to the learning-rate schedule: 40 fewer steps and 2.2 s, with validation loss above the baseline until the learning rate starts to decay, and much smaller final weights. Record #48 fixed which axis counts as a neuron: on the smear gate, a single weight vector, the wrong orientation normalized each entry by its own history, which is sign descent. Record #50 extended cautious decay to the Adam-trained parameters except scalars (20 steps, 1.1 s). The essay's explanations are the author's reasoning; the ANVIL optimizer of record #92 still combines a NorMuon-style row rescale with cautious decay.

Consequences. Muon became a general optimizer once its update size was tied to AdamW's and its weights were held by decay (Kimi K2 and GLM-4.5 later trained with it); the speedrun refinements then remove what orthogonalization leaves behind, unequal neuron update sizes and a decay that can raise the loss. The evidence is empirical, mostly single runs without error bars.