The NanoGPT speedrun

Context. The NanoGPT speedrun is a public experiment in how fast eight H100s can train a language model to 3.28 cross-entropy on the FineWeb validation set, the loss that Karpathy's llm.c reproduction of GPT-2 small (124M parameters) reached after 45 minutes and 10B tokens in May 2024. Through 92 records the time has fallen to 0.665 minutes (ANVIL2, August 2026). The rules make each record a controlled comparison: the training and validation token streams are fixed; a record must show that its mean validation loss over several runs is at most 3.28 at one-sided $p<0.01$ and must beat the previous record timed on the same hardware; and changes that only touch systems performance are exempt from the significance test, since they compute the same updates up to rounding. The target is a probability, so any valid probability model counts, evaluated at any context length: the model must assign at least $e^{-3.28\times10{,}485{,}760}$ to the first 10,485,760 validation tokens. A record may therefore validate with longer attention windows than it trains with. The model stays near 124M parameters, and time is the only quantity being minimized.

Core principle. Wall-clock time is steps times time per step, and almost every modelling idea is used in the same way: it lowers the loss reached at a fixed number of steps, and the number of steps is then reduced until the mean sits just below 3.28. A change is worth keeping when the time saved by the dropped steps exceeds the time it adds to each step. The ANVIL2 author estimates that at the current record one millinat (0.001 nats) of validation loss is worth about 120 to 250 ms of wall-clock. Because the total keeps shrinking, compare records by fractions: a second at 1.5 minutes is worth what ten seconds was at 15. Per-change credits are approximate: records often bundle several changes, and early ones announced on X were never split, so many numbers are the author's own ablation; ablations are not additive, which the ANVIL2 README handles by valuing every millinat at the same 164 ms chosen so that its table sums to the measured 34 s win over record #89; and the record is kept 1 to 2 millinats below 3.28 on purpose, so a change that saves time only by using up that margin does not count.

Six records account for most of the speedup. Most of the 45 minutes came out in six records that each removed 20 to 41% of the time; from the 3-minute mark on, the median record saves about 1%, a second or two. Record #2 (45 to 31.4 min) combined rotary embeddings with a tuned, higher AdamW learning rate and a warmup-stable-decay schedule. Record #3 (31.4 to 24.9) brought Muon, which replaces each hidden matrix's momentum update by its nearest semi-orthogonal matrix through five Newton-Schulz steps, at most $Tm/B\approx0.7\%$ extra FLOPs ($T=5$ iterations, width $m=768$, $B=524{,}288$ tokens per step), so nearly all of its lower loss per step turns into saved time. Record #5 (22.3 to 15.2) combined ReLU², QK-norm, zero-initialized output projections and padded embeddings; QK-norm bounds the attention logits whose growth caps the learning rate, and zero-initialized projections start every block as the identity. Record #9 (10.8 to 8.2) added a value residual, a shortcut from the embedding into every block, momentum warmup and a tanh logit softcap. Record #12 (7.2 to 5.03) replaced 1024-token chunks with FlexAttention over 64K-token packed sequences, masked to each document and a 1024-token window, skipping fully masked tiles. Its steps got slower (145 to 162 ms); the time came from needing 1875 steps instead of 3000, since each token now sees clean context from its own document, and the skipped tiles kept the attention work near twice that of 1024-token chunks. The record does not separate this from its switch to document-masked validation. ANVIL2 (#92, 1.126 to 0.665) bundles a new optimizer stack, sampled softmax, an 84.6M-row hashed n-gram table, full FP8, mixed-width attention and CUDA-graph capture. Its largest item, sampled softmax over a shared candidate set (8.4 s in its ablation), biases the training gradient; the rules allow it because only the full-vocabulary validation loss is graded.

Reading guide. The Muon note covers the optimizer and how it entered the speedrun; the Muon theory note reads the orthogonalized update as steepest descent under the spectral norm, as a trust-region step and as a spectral-norm constraint; and the note on refinements covers two years of small wins that each tune part of the same update (Polar Express polynomials, NorMuon's per-neuron rescale, cautious weight decay). The schedules note covers how much each token lowers the loss: warmup-stable-decay schedules, where the loss falls fastest during the decay, and batch-size ramps read through the critical batch size, which grows as the loss falls. The stability note covers what caps the learning rate in a small model and the fixes (QK-norm, zero-initialized projections, the logit softcap). The long-context note covers FlexAttention, document masking (#12) and window warmup (#13). The cross-layer note covers later layers reading earlier values or hidden states directly, from the value residual to MUDD connections (#81, #85), and the n-gram note covers hashed tables, from the bigram embedding of #62 to the table of #92. In both, a cheap path or lookup supplies, at almost no FLOPs, a feature a small model would otherwise spend layers or steps to build. Systems records (lower precision, PyTorch upgrades, overlapped communication, fused kernels, CUDA graphs) reduce time per step without changing the math; only FP8 formats and FlashAttention-3 have papers, and no note covers them.

What has no paper, and what the record can show. Much of the record rests on no paper: U-net skips, window warmup, smear, backout, paired-head attention, the mantissa buffer and ANVIL, among others, come from tweets, PRs and READMEs, and their evidence is the record itself. That evidence is strong in one respect: a change survives only by beating a well-tuned previous record under a significance test, which is Jordan's argument that a competitive benchmark solves the problem of new methods being compared against poorly tuned baselines. It is limited in scope: everything is tuned for a 124M model, under a billion tokens and one validation set, so a trick can win here and fail at scale; the README expects changes that impose extra structure, such as logit softcapping, not to transfer, while a scaled-up version of the October 2024 run reached GPT-2 XL's HellaSwag score at 1.5B parameters 2.5 times more cheaply than llm.c.