Gupta et al., 2018; Anil et al., 2020; Vyas et al., 2025

Adam rescales each coordinate of the gradient separately, so its step can only be stretched or shrunk along the coordinate axes. Full-matrix AdaGrad can rotate as well, but for a weight matrix with m rows and n columns its preconditioner has (mn)² entries, about 5.6×10¹² for one 768×3072 layer of GPT-2 small, so it is not run on networks.

Cheng and Zheng, 2026

Run full-batch gradient descent with a constant learning rate η on a neural network, and the sharpness λ rises until ηλ≈2 and then stays there while the loss keeps falling, although on a fixed quadratic any ηλ>2 diverges.

Zheng, 2026

In Muon, as in Adam, the momentum is a single exponential moving average (EMA) of past gradients with one fixed decay rate, 0.95 in the speedrun's Muon.

Schmied et al., 2025

In-context reinforcement learning trains a transformer offline on many tasks so that, on a new task, it improves from episode to episode by reading its own earlier episodes, with no weight updates.

Mongaras et al., 2024; Mongaras, Larson and 2025, 2026

Causal softmax attention makes each output a weighted average of all earlier values, so a decoder keeps every past key and value (the KV cache) and training cost grows with the square of the sequence length.

Schmied et al., 2025; Hauzenberger et al., 2026

A transformer decodes by attending to a KV cache that grows with every token, so the time and memory of each step grow with the context. That matters for a robot controller, which must answer at the rate of its sensors: 100 to 1,000 Hz, or under 10 ms per step, and 15 seconds of history at 1,000 Hz is a context of 15,000 steps.

Hauzenberger et al., 2026

To decode without recomputing the past, a transformer keeps the key and value of every earlier token, the KV cache, and reads all of it at every step, so memory and the time per token grow with the length of the output.

Ziegler et al., 2019; Huang et al., 2024; Ahmadian et al., 2024

Ziegler et al. (2019) set up reinforcement-learning fine-tuning of a pretrained language model from human preferences: a reward model trained on comparisons, a KL penalty to the starting model subtracted from the reward, and PPO taken over from game and robotics RL, with its learned value model, per-token advantages, clipping and several passes over each batch.

Shao et al., 2024; Guo et al., 2025

The usual way to fine-tune a language model with reinforcement learning is the KL-regularized objective of InstructGPT optimized with PPO. It trains a reward model on human comparisons and a value model next to the policy, each about as large as the policy itself.

Yu et al., 2025; Liu et al., 2025

After DeepSeek-R1, open attempts to reproduce R1-Zero ran GRPO with answer-checking rewards on open base models and ran into trouble. On Qwen2.5-32B a straightforward GRPO run reached 30% accuracy on AIME 2024, against 47% reported by DeepSeek for the same base model, with the policy's entropy collapsing early; other runs produced ever longer answers.

Zheng et al., 2025

In RL with verifiable rewards, a GRPO step samples a group of answers to each question, scores them, and then takes several gradient steps on that batch, one per minibatch.

Liu et al., 2025

RL recipes for reasoning built on GRPO stack many small techniques, and published recipes disagree about them. GRPO standardizes rewards within each prompt's group while REINFORCE++ standardizes over the batch; GRPO divides by the standard deviation while Dr. GRPO argues against it; GRPO averages the loss within each response while DAPO averages over all tokens of the batch.

Lambert et al., 2024

Post-training shapes most of what users see in a language model, and by 2024 the strongest recipes were closed, run over several rounds on millions of examples that were never released.

Lightman et al., 2024

A reward model that scores solutions chooses among samples at test time, and in RL the policy learns to please it, so its reliability limits both uses.

Cui et al., 2025; Wang et al., 2025

In RLVR the model's policy entropy drops sharply in the first steps and keeps falling to near zero unless something stops it, and validation accuracy stalls at about the same time.

Yue et al., 2025; Liu et al., 2025

RLVR is how the recent reasoning models are trained, and the hope behind it comes from games, where RL found strategies no human had shown it.

Snell et al., 2025

A language model can spend extra computation on a hard prompt in two ways. It can sample many answers and let a verifier choose among them, or it can write an answer, read it and revise it.

Gehring et al., 2025; Wei et al., 2025

In code a reward can be computed by running the program on tests. Earlier RL for code used that signal for single attempts, while the loops used in practice (write code, run it, read the error, repair) were built by prompting, and at an equal number of samples they rarely beat drawing independent solutions.

Schmied et al., 2026

Language models are being used as agents on the hypothesis that their knowledge and step-by-step reasoning let them act well after little interaction.

Steinberger, 2019; Steinberger et al., 2020

In a two-player zero-sum game with hidden information, such as poker, the target is a Nash equilibrium, and the standard way to compute one is counterfactual regret minimization (CFR).

Paulus et al., 2020; Paulus et al., 2021

Models with a discrete choice inside, such as a subset of words that explains a prediction, a latent graph or a parse tree, are trained by gradient descent on an expected loss 𝔼[L(X)], where the discrete random X ~ p_θ depends on parameters θ.

Qi et al., 2025

Reinforcement-learning (RL) fine-tuning of a language model runs two programs on the same weights. An inference engine generates the responses, and a training engine recomputes their token probabilities and takes the gradient.

Zhong et al., 2026

In reinforcement-learning (RL) fine-tuning of a language model, an inference engine generates the responses and a training engine computes the gradient, and the two give slightly different probabilities to the same tokens under the same weights because their kernels round differently.

Qi et al., 2026

PPO, the standard algorithm for reinforcement-learning (RL) fine-tuning of language models, descends from TRPO. TRPO keeps each new policy within a set divergence of the policy that collected the data, a constraint called a trust region.

Tian et al., 2026

GRPO for RL with verifiable rewards is usually run close to on-policy: the current model samples a batch of answers, the trainer takes a few gradient steps on it (four is a common choice), and then the model samples again.

Ritter et al., 2026

Policy-gradient methods such as PPO and GRPO assume that the training data come from the policy being trained, and in LLM post-training they rarely do.

Sheng et al., 2025

One step of RL post-training has two phases: rollouts generate trajectories, and the trainer updates the weights on them. Generation dominates: in math reasoning it took up to 83.1% of a step in the runs of DAPO, and lengths are so skewed that the 99th percentile can be ten times the median, so most GPUs sit idle while the last long trajectories finish.

Khatri et al., 2026

In pretraining, loss fitted against compute on small runs predicts the loss of large ones. RL for reasoning, now run at budgets such as the 100,000 GPU-hours reported for DeepSeek-R1-Zero, has had nothing similar.

Prime Intellect Team, 2025

Open-weight models trained with RL on verifiable rewards are now strong at math and code, but the machinery that trains them (the asynchronous trainer, the environments and their reward checks, the code sandboxes and the evaluations) has mostly stayed inside the groups that built it or in monolithic frameworks.

Stein, 1956; James and Stein, 1961; Efron and Morris, 1973

Let X ~ N_d(θ,I_d) be a single observation of an unknown mean θ∈ℝᵈ, to be estimated under total squared error. The maximum likelihood estimate (MLE) X is unbiased, has constant risk d, is minimax, and for d=1 is admissible.

Belkin, Hsu, Ma and Mandal, 2019; Belkin, Hsu and Xu, 2020

Neural networks, random-feature models and kernel machines are trained to zero training error, often on noisy labels, and still predict well, contrary to the textbook U-shaped curve of test risk against model size.

Bartlett, Long, Lugosi and Tsigler, 2020; Tsigler and Bartlett, 2023

Deep networks fit noisy labels exactly and still predict well. Bartlett, Long, Lugosi and Tsigler (BLLT, 2020) take the simplest version of the puzzle and ask when, in linear regression with a general covariance Σ, the minimum-norm interpolant of noisy data has excess risk near zero.

Muthukumar, Vodrahalli, Subramanian and Sahai, 2020

Deep networks fit noisy labels and still predict well, while the variance of least squares diverges as d approaches n.

Hastie, Montanari, Rosset and Tibshirani, 2022

Deep networks fit noisy data exactly, still predict well, and show "double descent" in model size (Belkin et al. 2018).

Cheng and Montanari, 2024

Consider ridge regression on n samples of d-dimensional covariates with covariance Σ and true coefficients β. Random-matrix analyses of ridge give sharp formulas only for n≍d and well-conditioned Σ, with additive errors that are uninformative once the risk tends to zero; Bartlett-Long-Lugosi-Tsigler and Tsigler-Bartlett handle any dimension but only up to constants.

Koehler, Zhou, Sutherland and Srebro, 2021; Zhou, Koehler, Sutherland and Srebro, 2021

Bartlett, Long, Lugosi and Tsigler (2020) proved benign overfitting for the minimum ℓ₂-norm interpolator through its explicit formula, and uniform convergence seemed unable to (Nagarajan and Kolter, and several impossibility results).

Liang and Rakhlin, 2020; Belkin, Rakhlin and Tsybakov, 2019

Interpolating rules were widely expected to overfit: classical theory in an RKHS trades bias against variance through a tuned ridge and gives ridgeless kernel regression no guarantee, and the classical interpolator, 1-nearest-neighbor, is inconsistent (asymptotic risk up to twice Bayes).

Mallinar, Simon, Abedsoltan, Pandit, Belkin and Nakkiran, 2022

Benign-overfitting theory reaches Bayes risk only with dimension growing with n or a very slowly decaying spectrum. Networks trained to interpolation on fixed-dimension data behave differently: their test error neither diverges nor reaches Bayes risk as n→∞, and it grows with the label noise.

Soudry, Hoffer, Nacson, Gunasekar and Srebro, 2018

On underdetermined least squares, gradient descent from the origin picks the minimum-norm interpolant. Logistic and cross-entropy losses on separable data have no finite minimizer, so the loss reaches zero only as ‖w(t)‖→∞, and the question is whether the direction w(t)/‖w(t)‖ converges and to what.

Bordelon et al., 2020; Maloney et al., 2022

Test loss falls as a power law in data and in parameters, and two papers explain the exponents; their spectral results reduce to one equation.

Bahri et al., 2021

Test loss of trained networks falls as a power law in D and P (Hestness et al., Kaplan et al.), but it was unknown what sets the exponents and whether the D and P exponents are related.

Hoffmann et al., 2022

Kaplan et al. (2020) fit power laws for language-model loss and concluded that a growing budget should go mostly into model size. At a compute budget C, write Nₒₚₜ(C) and Dₒₚₜ(C) for the best allocation, and Nₒₚₜ, Dₒₚₜ when the budget is clear.

Lin et al., 2024; Paquette et al., 2024

Chinchilla measured that the compute-optimal model size grows like the square root of compute, with a fitted law that has no variance term although textbook variance grows with model size.

Ben Arous et al., 2021; Damian et al., 2023

Online SGD estimating an unknown unit vector θ∈ℝ^N from a stream of samples starts from a random point x on the unit sphere. Its correlation m=x·θ with the truth is about N^(-1/2), so it starts essentially on the equator m=0, where per-sample gradient noise dwarfs the population gradient.

Ba et al., 2022; Damian et al., 2022

A two-layer network with a frozen random first layer is a random-features model. Like the NTK and other fixed kernels, it is essentially linear in the proportional regime, and needs n≳dᵖ samples for a degree-p polynomial even when the polynomial depends on few directions.

Abbe et al., 2022

Let f_*(x)=h_*(x_I) be a function on the hypercube {±1}ᵈ that depends only on an unknown set I of P coordinates, with x_I the restriction of x to I. Expand h_* in the monomials χ_S(z)=∏_(i∈S)zᵢ of its argument z∈{±1}^P, the Fourier basis of the cube.

Barak et al., 2022

Let x be uniform on {±1}ⁿ (here n is the input dimension) and S an unknown set of k coordinates.

Garg et al., 2022

Language models pick up a task from a few examples in the prompt, and it was unclear whether they learn from the examples or retrieve a task seen in training.

von Oswald et al., 2023; Ahn et al., 2023; Zhang et al., 2024

Transformers trained on random linear-regression prompts predict about as well as least squares (Garg et al.), which says nothing about mechanism.

Bai et al., 2023

Garg et al. showed that a transformer trained on random regression prompts predicts like least squares, and von Oswald et al. and Akyürek et al. built layers that take gradient steps on the prompt's squared loss.

Xie et al., 2022; Raventós et al., 2023

GPT-3 learns a task from a few input-output examples in its prompt with no gradient update, although nothing in next-token pretraining asks for this and a prompt of concatenated unrelated examples is an unnatural, low-probability document.

Olsson et al., 2022

Language models predict later tokens in a context better than early ones. The earlier circuits paper (Elhage et al., 2021) found induction heads, a two-head circuit that performs the completion [A][B]…[A]→[B], in two-layer attention-only transformers.

Jacot et al., 2018; Chizat et al., 2019

At initialization a wide network is a Gaussian process (the NNGP correspondence), but that says nothing about training a loss that is nonconvex in the parameters, and by 2018 heavily over-parameterized networks were seen reaching zero training loss while their parameters barely moved.

Mei et al., 2018

The population risk of a two-layer network is nonconvex in the weights. Mei et al. send the number N of hidden units to infinity with the 1/N output normalization and follow the empirical distribution of the neurons, identifying each neuron with its parameter vector.

Yang and Hu, 2021

By 2020 wide networks had two infinite-width limits, NTK and mean field, and the standard parametrization fit neither.

Song et al., 2021

SMLD (Song and Ermon, 2019) and DDPM (Ho et al., 2020) both corrupt data with Gaussian noise and learn to undo it. SMLD learns the score of the noised data by score matching at a sequence of noise levels and samples with Langevin dynamics; DDPM samples by a reverse Markov chain whose network predicts the added noise ε.

Chen et al., 2023

Song et al. (2021) gave diffusion models a standard procedure: learn the score of the data noised by an Ornstein-Uhlenbeck (OU) process by L² regression, plug it into the reverse-time SDE, and discretize.

Montanari, 2023

Diffusion samplers were derived by time reversal, by noising the data with an SDE, reversing it and learning the score. Stochastic localization came from elsewhere.

Kadkhodaie et al., 2024

A diffusion model is a denoiser, a map from a noisy image y to an estimate of the clean one, trained by least squares at every noise level. Over all functions, the objective has a useless exact minimizer.

Elhage et al., 2021

Circuit-level reverse engineering had worked for vision models because many conv-net neurons are individually interpretable, so one can read the weights between two of them.

Elhage et al., 2022

Many neurons in trained networks are polysemantic. The leading explanation was the superposition hypothesis: a network represents more features than it has dimensions by giving each feature its own direction and tolerating interference between directions that are not orthogonal.

Candès and Tao, 2005

Let f∈ℝⁿ be a plaintext and A an m×n coding matrix with m>n. The sender transmits Af∈ℝᵐ, and an adversary corrupts an unknown set of entries arbitrarily: the receiver sees y=Af+e, where the error vector e∈ℝᵐ is nonzero only on the corrupted entries.

Bricken et al., 2023

Elhage et al. (2021) read attention-only transformers off their weights but stopped at the MLP, whose neurons are mostly polysemantic. Toy Models of Superposition (Elhage et al., 2022) explained polysemanticity as superposition (a layer storing more features than it has neurons, as non-orthogonal directions) and proposed two remedies: train models without it, or find an overcomplete basis after training.

Zhang et al., 2017

Classical theory explains a small train-test gap through the capacity of the hypothesis class, the stability of the algorithm, or regularization, yet networks with far more parameters than training images generalize, and it was unclear which of these was at work.

Bartlett et al., 2017

Zhang et al. showed that standard networks fit random labels, so any bound that depends only on the architecture and the sample size is vacuous.

Nagarajan and Kolter, 2019

After Zhang et al., the program was to apply uniform convergence only to the networks SGD actually reaches, with complexity measured by norms of the learned weights, margins or PAC-Bayes posteriors.

Dziugaite and Roy, 2017; Lotfi et al., 2022

A uniform convergence bound is no smaller than what the class can fit, and Zhang et al. showed that networks which generalize on real labels also fit random ones.

Ouyang et al., 2022

Next-token prediction on web text differs from "do what this user asks, truthfully and without harm", and scale alone does not close the gap: a larger GPT-3 still ignores instructions, makes things up and needs careful prompting.

Korbak et al., 2022; Rafailov et al., 2023; Azar et al., 2024

RLHF fits a reward to human comparisons and runs PPO against it with a KL penalty to the supervised model, which means several large models in memory, sampling inside the training loop and a hard-to-train value function.

Gao et al., 2023

RLHF optimizes a learned reward model, an imperfect proxy for human judgment, and pushing it too hard makes outputs worse.

Lei et al., 2018; Romano et al., 2019

Regression had confidence intervals for coefficients, each resting on a correct model, and almost nothing for the question a user asks: how far can the next response be from the prediction?

Tibshirani et al., 2019

Every conformal guarantee rests on exchangeability of training and test points, and the most common way that fails is covariate shift. Under covariate shift the test inputs come from P̃_X instead of P_X, and P_(Y|X) is unchanged.

Gibbs and Candès, 2021

Split conformal prediction gives a set Ĉ with ℙ(Y∈Ĉ)≥1-α, for a chosen miscoverage α, only when calibration and test points are exchangeable.

Lipton et al., 2018

Suppose a classifier is trained when 0.1% of cases are positive and deployed when 5% are; every borderline case is then scored against an obsolete prior.

Elkan and Noto, 2008; Kiryo et al., 2017

Many data sets list positives and nothing else: a curated database names the proteins that belong in it, and everything else mixes true negatives with positives nobody has labeled yet.

Lakkaraju et al., 2017

To ask whether a model would make better bail decisions than judges, one needs outcomes for the defendants the model would release, but outcomes exist only for those the judge released.

Dudík et al., 2011; Swaminathan and Joachims, 2015

A deployed policy picks one action per context and sees only that action's reward, and the questions are how a different policy would have done and which policy to deploy.

Hardt et al., 2016; Brückner et al., 2012

A classifier that decides about people, spam or fraud gets gamed: the other side changes cheap attributes to cross it. Hardt et al. (2016) set up a Stackelberg game, in which a jury commits to a classifier and each contestant best-responds under a cost the jury knows, and ask for a learner whose accuracy after the response is within ε of the best possible.

Perdomo et al., 2020; Miller et al., 2021

A deployed model changes the data it will next be trained on: a credit score changes who defaults, and a fraud model changes what fraud looks like.

Zrnic et al., 2021

Strategic classification and performative optimality fix the order of play: the decision-maker deploys a rule and agents best respond instantly.

Elkan, 2001; Saerens et al., 2002

By 2001 cost-sensitive learning had many algorithms but loose foundations: published cost matrices mixed baselines, and rebalancing the training set was a common practice with no formula for how much.

Fithian and Hastie, 2014

In rare-positive problems (clicks, fraud, disease) the information sits mostly in the positives while the compute scales with the total. Epidemiology's answer, case-control sampling, keeps all cases and a few controls, fits logistic regression, and subtracts the log sampling ratio from the intercept (Anderson, 1972; Prentice and Pyke, 1979).

Guo et al., 2017; Niculescu-Mizil and Caruana, 2005

Learners were mostly compared on accuracy and ROC area, but a cost threshold p* applied to a model's predicted probability p̂(x) makes the right decision only if p̂(x) is an accurate probability.

Menon et al., 2021

When a few classes dominate the training data, plain cross-entropy favors them, and the usual target becomes the balanced error. Recent deep-net fixes either divide each class's logit by its weight norm after training or change the loss with per-class margins or weights; they work on benchmarks, but the margin and normalization fixes were not derived from the target's Bayes rule, which this paper writes down.

Jordan et al., 2024–2026

The NanoGPT speedrun is a public experiment in how fast eight H100s can train a language model to a cross-entropy of 3.28 on the FineWeb validation set, the loss that Karpathy's llm.c reproduction of GPT-2 small (124M parameters) reached after 45 minutes and 10B tokens in May 2024.

Jordan et al., 2024; Bernstein and Newhouse, 2024; Bernstein, 2025; Amsel et al., 2025

In 2024 the optimizers that beat AdamW per step, such as Shampoo and SOAP, lost on wall-clock because their matrix inverse roots are expensive and need float32.

Kovalev, 2025; Chen et al., 2025

Muon orthogonalizes the momentum of each hidden matrix, replacing its singular value decomposition USVᵀ by UVᵀ, and, at scale, adds decoupled weight decay, W←(1-ηλ)W-η UVᵀ with learning rate η and decay rate λ, which Moonshot AI found essential for long runs.

Liu et al., 2025; Li et al., 2025; Chen et al., 2025; Srivastava, 2025

Muon (see the Muon post in this section) replaces a hidden matrix's momentum M by its polar factor UVᵀ, which keeps the singular vectors and sets every singular value to 1, so every singular direction of the update gets the same step.

Schaipp et al., 2025; McCandlish et al., 2018

Under warmup-stable-decay (WSD, or trapezoid), the loss plateaus and then drops sharply at the cooldown, which had been explained by nonconvex geometry.

Wortsman et al., 2023

Very large transformers showed failures never seen at small scale: attention logits that grow until the softmax collapses (ViT-22B) and output logits that drift away from the log-probabilities they represent (PaLM).

Dong et al., 2024; records #12-#13

FlashAttention made exact attention fast by fusing the two matmuls and the softmax into one tiled kernel that never writes the T×T matrix of attention scores of a T-token sequence to memory.

Zhou et al., 2024; Xiao et al., 2025

In a standard pre-norm transformer, every layer reads one residual stream and writes its output back into it, so all communication between layers goes through a single running sum.

Cheng et al., 2026

Mixture-of-experts models scale parameters without scaling compute by sending each token through a few of many expert MLPs, a form of sparsity in computation.

Eight published transaction and banking foundation models.

What CPC, replaced-event detection and temporal contrast learn, and what they miss.

Memory, neighbors and honest evaluation on timestamped graphs; serving in real time.

Graphs built from relational tables, graph pretraining, and graph foundation models.

Fraud GNNs against tree baselines, edge-level crime patterns, and the trouble with labels.