Sam Babichenko

Reading notes

Threads

Benign overfittingGeneralization boundsLazy training and feature learningHigh-dimensional SGDScaling lawsIn-context learningSuperpositionDiffusion modelsNanoGPT speedrunLearning from human preferencesMissing and selective labelsCost-sensitive classification and calibrationConformal prediction and distribution shiftStrategic and performative predictionFoundation models for transaction data Shrinkage toward a common point The data spectrum sets an effective ridge Noise absorbed by many weak directions Scale decides whether features move Implicit bias of the optimizer Abrupt transitions at a critical ratio Averaging out gradient noise Exponential tilting Importance weighting for distribution shift Separating probability estimates from decision thresholds Feedback between a model and its data Fixed scales for stable training

Shrinkage toward a common point

When many quantities are estimated at once, the raw estimates are too spread out, so pulling them all toward a common point by an amount learned from the data lowers the total error. Ridge, empirical Bayes, partial pooling, soft thresholding and a trained denoiser all do this, keeping the components that stand above the noise and shrinking the rest.

  1. Benign overfittingStein's paradox and the James-Stein estimatorBecause ∥X∥2\|X\|^2 exceeds ∥θ∥2\|\theta\|^2 by about dd on average, the best factor is about 1−d/∥X∥21-d/\|X\|^2, and the James-Stein version beats the raw observation at every θ\theta once d≥3d\ge3.Under a normal prior the same rule is the partial-pooling posterior mean with the prior's spread estimated from the data, and with orthonormal design ridge at penalty λ\lambda is the Bayes rule for prior variance σ2/λ\sigma^2/\lambda, so James-Stein is ridge with λ\lambda tuned on the data.
  2. Benign overfittingBenign Overfitting in Linear RegressionIn the minimum-norm interpolant each weak tail direction is shrunk by the factor λi/∑j>kλj\lambda_i/\sum_{j>k}\lambda_j, as ridge would shrink it, while each strong head direction has the least-squares error of about 1/n1/n.
  3. In-context learningTransformers Learn In-Context by Gradient DescentA trained linear-attention layer keeps the fraction Nλi/((N+1)λi+trΛ)N\lambda_i/((N+1)\lambda_i+\mathrm{tr}\Lambda) of the signal in eigendirection ii, a ridge-like shrinkage fixed at training time, which is why its error keeps a 1/M1/M term even on noiseless prompts.
  4. In-context learningIn-context learning as implicit Bayesian inferenceThe transformer's switch from the Bayes rule for the tasks it saw to ridge is the switch from using the empirical distribution of past tasks as a prior to using a smooth prior fitted to them, and data scale alone does not bring it about; task diversity has to grow too.
  5. In-context learningTransformers as StatisticiansA transformer can run the Lasso because soft thresholding, the shrinkage step of proximal gradient, is a difference of two ReLUs, which a token-wise MLP computes exactly.For prompts with mixed noise levels it fits ridge at several penalties and keeps the one that validates best, which is nearly Bayes-optimal at an extra cost of order (log⁡K/N)1/3(\log K/N)^{1/3}.
  6. Diffusion modelsGeneralization in diffusion modelsA bias-free denoiser computes f(y)=∑kλk(y)⟨y,ek(y)⟩ek(y)f(y)=\sum_k\lambda_k(y)\langle y,e_k(y)\rangle e_k(y), shrinkage in a basis adapted to the noisy image that keeps the coefficients above the noise and suppresses the rest.
  7. Diffusion modelsStochastic localizationA diffusion sampler is an SDE whose drift is the posterior-mean denoiser m(y;t)m(y;t), so sampling needs only a good shrinkage estimate at each noise level and does not need the full posterior.

Related threads

Elsewhere on this site

  • Partial Pooling for Estimating Interaction Effects in Factor ModelsThe partial-pooling post pools each school's estimate toward a shared mean with weights set by its noise and the spread between schools, then applies the same pooling to stock-characteristic interactions
  • Decision MeshThe Decision Mesh shrinks each finer correction toward zero under a prior whose variance at each depth is estimated by EM, so fine detail needs more evidence than coarse

The data spectrum sets an effective ridge

In high-dimensional ridge or minimum-norm regression, the many weak directions of the covariance act together like an added ridge penalty. One fixed-point equation gives that penalty from the eigenvalues; directions with eigenvalues above it are learned, the rest are lost to bias, and the test error is read off that split. When a result depends on the dimension, check which effective rank it refers to.

  1. Benign overfittingSurprises in High-Dimensional Ridgeless Least Squares InterpolationAs p/n→γp/n\to\gamma, the minimum-norm interpolant behaves like ridge on the population covariance with a penalty τ\tau fixed by Tr Σ(Σ+τ)−1=n\mathrm{Tr}\,\Sigma(\Sigma+\tau)^{-1}=n, and directions with eigenvalue well above τ\tau are learned.
  2. Benign overfittingBenign Overfitting in Linear RegressionThe tail's Gram matrix is close to a multiple of the identity once the effective rank rkr_k exceeds a constant times nn, so the tail's total energy ∑i>kλi\sum_{i>k}\lambda_i acts as a ridge on the top kk directions.
  3. Benign overfittingKernel interpolationIn high dimension a curved kernel's training matrix is a linear kernel plus γI\gamma I, the gap between the kernel's value on the diagonal and its linearization, so kernel interpolation is in effect ridge regression with a linear kernel.
  4. Benign overfittingDimension free ridge regressionWith the deterministic equivalent, interpolation is benign exactly when few directions sit above the effective penalty and the signal lies in them; a pure power law i−αi^{-\alpha} has too thin a tail, so the variance never vanishes.
  5. Benign overfittingBenign, Tempered, or CatastrophicRead as a conservation law, the equation caps the summed learnabilities at nn, and interpolation is benign when that total is spread thinly over many modes and catastrophic when it concentrates on nn modes that each fit their own noise.
  6. Scaling lawsSpectrum-dependent learning curvesEach kernel eigenmode is fitted as ridge on its own eigenvalue with penalty κ/p\kappa/p, and on a power-law spectrum the learned cutoff moves in proportion to the sample size, which gives power-law learning curves.
  7. Scaling laws4+3 Phases of Compute-Optimal Neural Scaling LawsFor SGD on a power-law spectrum the cutoff moves with training: directions with eigenvalue above 1/(Neffγ)1/(N_{\rm eff}\gamma) are learned, and each direction SGD has not reached keeps its full share of the loss.
  8. Generalization boundsSpectrally normalized margin boundsThe width correction in the spectrally normalized margin bound is at most width⋅r0(A⊤A)\sqrt{\text{width}\cdot r_0(A^\top A)}, with r0r_0 the same stable rank that measures tail mass in the benign-overfitting papers.
  9. Benign overfittingUniform convergence of interpolatorsUniform convergence for interpolators uses the same head-tail split: the head costs a factor 1−O(rank Σ1/n)1-O(\sqrt{\mathrm{rank}\,\Sigma_1/n}), and the complexity term is computed on the tail covariance only.

Related threads

Noise absorbed by many weak directions

Fitting noise exactly is harmless when the signal sits in a few strong directions and very many weak directions absorb the noise, each taking a small share that carries little weight under the test distribution. The same split decides whether adding features helps and which features a network keeps.

  1. Benign overfittingHarmless interpolation of noisy data in regressionAn interpolator must put the noise somewhere, the cheapest place is spread thinly over many weakly weighted directions at a cost of about σ2n/d\sigma^2n/d, and that spreading also moves part of the signal onto aliases, features that agree with the true one on the training points, unless a few strong directions hold it.With spiked feature weights the noise term is about σ2(s/n+n/d)\sigma^2(s/n+n/d), so consistency needs the strong features to be few (s/n→0s/n\to0) and the weak ones to be many (d/n→∞d/n\to\infty).
  2. Benign overfittingBenign Overfitting in Linear RegressionBenign overfitting needs o(n)o(n) strong directions, a tail that is heavy and flat enough that the unimportant directions greatly outnumber nn, and signal that sits in the head.
  3. Benign overfittingSurprises in High-Dimensional Ridgeless Least Squares InterpolationWith isotropic features overparametrization never beats γ→0\gamma\to0, but when the spectrum is spikes plus a bulk and the signal sits in the spikes, the weak directions add about γIn\gamma I_n, which acts as a ridge penalty, and the best explicit penalty tends to zero.
  4. Benign overfittingDouble descentWith random, individually weak features each one adds variance and removes little misspecification, while features chosen in order of strength put the optimum below nn and make interpolation useless.
  5. Generalization boundsLimits of uniform convergenceThe classifier is a simple part that carries the signal plus a memorizing part spread over D≫mD\gg m nearly orthogonal noise directions, which fits every training label and changes predictions on new points very little.
  6. Scaling lawsSpectrum-dependent learning curvesNormalized mode errors are ordered opposite to the eigenvalues at every sample size, so a target whose power sits in the top modes is learned faster.
  7. SuperpositionToy Models of SuperpositionWith fewer dimensions than features, a model gives dimensions to its most important features and, as features get sparser, packs the less important ones into superposition, least important first.

Related threads

Scale decides whether features move

A wide network whose weights barely move is a linear model on fixed random features, which makes it a kernel method, easy to analyze and limited in the same ways as kernel methods. Whether features move is set by scales such as the output multiplier, the initialization and the step size, and moving features are what beat kernels on targets that depend on a few hidden directions.

  1. Lazy training and feature learningThe neural tangent kernelAny model becomes lazy, its tangent kernel effectively frozen, when its output scale grows and its initial output is cancelled; for a width-mm network a 1/m1/\sqrt m output normalization is lazy in the limit and the mean-field 1/m1/m is not.Linearized VGG-11 and ResNet-18 reach 61.7% and 56.7% on CIFAR-10 against about 90% trained normally, though the linearized models also fit their training sets poorly, so the gap mixes a weaker function class with harder optimization.
  2. Lazy training and feature learningMaximal update parametrizationUnder the standard parametrization a stable learning rate must shrink like 1/width1/\text{width}, which freezes the hidden features; μ\muP (the maximal-update parametrization) rescales the output layer and the first-layer gradients so every layer's features move by order one.
  3. Benign overfittingSurprises in High-Dimensional Ridgeless Least Squares InterpolationRandom features with no linear part have a covariance within a polylog factor of the identity, so their limiting risk is that of an isotropic linear model.
  4. High-dimensional SGDFeature learning from one gradient stepOne first-layer gradient step of order one leaves the features equivalent to a linear model plus Gaussian noise, so the error cannot fall below the target's nonlinear part, while a step of order N\sqrt N moves each neuron along the target and beats every kernel with n≍dn\asymp d samples.
  5. High-dimensional SGDThe merged-staircase propertyAny linear method's prediction lies in a fixed span of at most min⁡(n,q)\min(n,q) functions, so it needs dkd^k samples on staircase targets (whose terms can be reached one new coordinate at a time) that SGD with moving features learns with O(d)O(d).
  6. High-dimensional SGDLearning sparse parities with SGDThe tangent kernel of a width-2O(k)2^{O(k)} network is too small an embedding to fit a kk-sparse parity, so the small networks that do learn parities must be moving their features.
  7. Lazy training and feature learningMean-field limits of two-layer networksWith 1/N1/N normalization the features move, and SGD follows a PDE over the distribution of neurons whose risk landscape is convex and does not depend on the width.

Related threads

Implicit bias of the optimizer

With more parameters than data, many solutions fit the training set, and the one you get is chosen by the optimizer, its starting point and when it stops. That choice acts as an implicit regularizer that appears nowhere in the objective, and it often decides generalization.

  1. Generalization boundsUnderstanding deep learning requires rethinking generalizationSGD started at zero stays in the row space of the data, so on a linear model it returns the minimum-norm interpolant, which reaches 1.2% test error on raw MNIST with no regularization.
  2. Benign overfittingDouble descentPast the interpolation threshold every model fits the data, so the test risk is set by which interpolant is chosen, and the minimum-norm one spreads the noise over more directions as features are added, which gives the second descent.
  3. Benign overfittingThe implicit bias of gradient descentGradient descent on logistic or exponential loss over separable data converges in direction to the max-margin SVM with no penalty, at rate 1/log⁡t1/\log t, which is why training past zero error can keep improving the margin.
  4. Benign overfittingHarmless interpolation of noisy data in regressionRidge is itself a minimum-norm interpolant on a design with nn extra fake features, one per training point, which absorb noise and vanish at test time.
  5. Cost-sensitive classification and calibrationTemperature scaling and calibrationOnce nearly every training point is classified correctly, the loss keeps falling by scaling up the logits, the growing weight norm of Soudry et al., and that is how modern nets become overconfident.
  6. Cost-sensitive classification and calibrationLogit adjustmentWeight normalization for imbalanced classes works only because SGD with momentum happens to make classifier norms track the class priors, and it fails under Adam at comparable balanced error.
  7. NanoGPT speedrunConvergence theory for MuonAny bounded update with decoupled weight decay λ\lambda drives the weights into a ball of radius b/λb/\lambda and keeps them there, so for Muon the weight decay setting is a hard spectral-norm constraint.
  8. Learning from human preferencesReward model overoptimizationAcross KL penalties from 0.01 to 0.5, gold score (from a larger reward model standing in for human judgment) traces one curve against KL, so the penalty's only effect on gold is where along that curve training stops.

Related threads

Abrupt transitions at a critical ratio

Many abrupt changes in training, a loss that sits flat and then drops or a behavior that switches on, come from a slowly changing ratio crossing a critical value, often signal against noise. When a curve jumps, finding that ratio tells you what to change to move the jump.

  1. High-dimensional SGDThe information exponentOnline SGD's overlap with a hidden direction grows at the rate signal2^2/noise, which is tiny near a random start, so escaping takes about dk⋆−1d^{k^\star-1} samples for information exponent k⋆k^\star (the order of the first nonzero Hermite term) and almost all the samples are used up before the overlap grows.
  2. High-dimensional SGDLearning sparse parities with SGDOn sparse parities the relevant weights grow steadily from the start while the loss stays flat, and the loss drops abruptly once they dominate the irrelevant ones.
  3. High-dimensional SGDFeature learning from one gradient stepOne large first-layer step creates an outlier singular value aligned with the target only once the step's signal strength passes the edge of the random spectrum, an outlier separating from the bulk.
  4. NanoGPT speedrunLearning-rate schedules and batch sizeBelow the noise scale tr⁡Σ/∣G∣2\operatorname{tr}\Sigma/|G|^2, the batch at which gradient noise equals signal, a bigger batch gives proportionally more progress per step; above it the gain saturates and linear learning-rate scaling breaks down.
  5. In-context learningIn-context learning as implicit Bayesian inferenceAt dimension 8, below about 2142^{14} pretraining tasks more data per task pulls the transformer toward the Bayes rule for the tasks it saw, and above 2152^{15} toward ridge, with the threshold growing roughly linearly in dimension.
  6. In-context learningInduction headsIn every model with more than one layer, most of the in-context learning score forms in one short window early in training, the same window in which induction heads form and the loss curve has its only non-convex stretch.
  7. SuperpositionToy Models of SuperpositionAn extra feature incurs interference only when it is active together with another, so superposition is better once features are sparse enough; for two features in one dimension the antipodal pair beats dropping the second exactly when its density is below 2r/(1+r)2r/(1+r).
  8. Benign overfittingDouble descentTest error peaks where the number of features equals the sample size, because the variance is an inverse-Wishart trace that is finite only from p=n+2p=n+2.
  9. Strategic and performative predictionPerformative predictionRepeated retraining converges when εβ/γ\varepsilon\beta/\gamma, sensitivity times smoothness over strong convexity, is below one, and the counterexamples show the threshold is sharp, since above it the stable point repels.
  10. Learning from human preferencesReward model overoptimizationBelow about 2,000 distinct comparisons a reward model stays near chance and gives no gold gain under optimization, whatever its size, and extra epochs on the same comparisons do not help.

Related threads

Averaging out gradient noise

A stochastic gradient carries noise that does not vanish at the optimum, so at a fixed step the iterate keeps jittering by an amount proportional to the step. Bigger batches, momentum, step-size decay and smoothing each reduce that noise at a cost of their own, and knowing which limit is active tells you which of them to change.

  1. NanoGPT speedrunLearning-rate schedules and batch sizeAt a constant step the last iterate fluctuates around the optimum in proportion to the step, and decaying the step at the end makes it behave like an average, which produces the sharp loss drop of a warmup-stable-decay cooldown.Steps and examples trade on a hyperbola, training at the critical batch costs twice the minimum of each, and the critical batch grows during a run as the gradient shrinks.
  2. NanoGPT speedrunConvergence theory for MuonWithout momentum a fixed-radius step keeps a noise floor that only a larger batch lowers; momentum averages about 1/α1/\alpha gradients and cuts the floor by α\sqrt\alpha, at the cost of a staleness term 2Lη/α2L\eta/\alpha.
  3. Scaling laws4+3 Phases of Compute-Optimal Neural Scaling LawsSGD's expected loss is the gradient-descent loss plus a term from noise injected while the loss is still of order one, so SGD noise matters only when the kernel decays more slowly than the gradient-descent loss.
  4. High-dimensional SGDThe information exponentSmoothing the loss over a cap of radius λ\lambda shrinks gradient noise faster than the squared signal up to λ=d1/4\lambda=d^{1/4}, which lowers the sample cost from dk⋆−1d^{k^\star-1} to the correlational-query lower bound dk⋆/2d^{k^\star/2}.
  5. Lazy training and feature learningMean-field limits of two-layer networksAdded noise turns the mean-field objective into a strictly convex free energy with one fixed point to converge to, at a cost in risk that falls like 1/β1/\beta as the temperature 1/β1/\beta goes to zero.
  6. Conformal prediction and distribution shiftAdaptive conformal inferenceAdaptive conformal inference updates its coverage level like constant-step SGD tracking a moving optimum, and its step size γ\gamma trades lag against jitter in the same way.

Related threads

Exponential tilting

The distribution that maximizes expected reward minus β\beta times its KL divergence to a base distribution is the base reweighted by er/βe^{r/\beta}, called its exponential tilt. The same identity defines the RLHF target, the Gibbs fixed point of noisy training and Bayesian posteriors, so in each case the practical question is how far the tilt moves you from the base and what keeps it finite.

  1. Learning from human preferencesDirect preference optimizationWithout the KL term the best policy puts all its mass on one sequence; with it the target is πrefer/β/Z\pi_{\rm ref}e^{r/\beta}/Z, which keeps mass exactly where the reference model puts it and moves toward the best sequences as β\beta falls.Taking logs of the tilt writes the reward as βlog⁡(π/πref)\beta\log(\pi/\pi_{\rm ref}) plus a normalizer that cancels when two responses are compared, which lets DPO fit preferences by supervised learning on a fixed dataset.IPO tilts by the win rate, which lies in [0,1][0,1], so log ratios to the reference span at most 1/τ1/\tau even when preferences are deterministic, whereas under DPO's logit link they grow without bound.
  2. Learning from human preferencesReward model overoptimizationBest-of-nn reweights each response by nF(r)n−1nF(r)^{n-1}, at a KL of log⁡n−(n−1)/n\log n-(n-1)/n, and best-of-1000 reaches 94% of the gain that the exponential tilt, the best policy at that KL, would get on a Gaussian score.
  3. Learning from human preferencesInstructGPTInstructGPT's KL penalty is there to keep the policy where the reward model can be trusted, and in its sweep only β\beta between 0.01 and 0.02 did well.
  4. Generalization boundsNonvacuous PAC-Bayes boundsIn a PAC-Bayes bound the KL from posterior to prior is the complexity term, and a Gaussian prior centered at the initialization makes it grow with how precisely the weights are specified and how far they move in training.
  5. Lazy training and feature learningMean-field limits of two-layer networksNoisy training minimizes risk plus entropy, and the unique minimizer is the Gibbs measure, proportional to e−βΨe^{-\beta\Psi}.
  6. In-context learningIn-context learning as implicit Bayesian inferenceIn-context prediction is a posterior average over latent concepts, each weighted by its prior times enrn(θ)e^{n r_n(\theta)}, the per-example log likelihood ratio to the true concept, so the prompt selects the concept once that ratio is negative for every other.
  7. Diffusion modelsStochastic localizationObserving Yt=tx+WtY_t=tx+W_t turns the prior into the tilt μ(dx)e⟨Yt,x⟩−t∥x∥2/2\mu(dx)e^{\langle Y_t,x\rangle-t\|x\|^2/2}, so a diffusion sampler runs through a sequence of ever sharper tilted posteriors.

Related threads

Elsewhere on this site

  • Risk Aversion, EntropicallyA risk-averse player in the risk-sensitive version of the games minimizes θ−1log⁡E eθC\theta^{-1}\log\mathbb E\,e^{\theta C} and judges deviations under its belief reweighted by eθCe^{\theta C}, the same tilt applied to costs, while its beliefs stay those of a risk-neutral player

Importance weighting for distribution shift

When training data come from a different distribution than deployment, weight each example by the likelihood ratio between the two, then control the variance that large ratios bring. Shifted inputs, shifted base rates, partly labeled positives, logged decisions and subsampled negatives all take this form, and the same fixes for the variance recur in each case, namely clipping, control variates, variance penalties and keeping some exploration.

  1. Conformal prediction and distribution shiftConformal Prediction Under Covariate ShiftUnder covariate shift each calibration point is weighted by w(xi)=dP~X/dPXw(x_i)=d\tilde P_X/dP_X, and the test point's own weight sits on +∞+\infty, so a test point far from the training data gets an infinite band, which correctly signals that the method cannot extrapolate to it.Reweighting reduces the effective sample size to ∥w∥12/∥w∥22\|w\|_1^2/\|w\|_2^2, and in practice the ratio comes from the odds of a classifier trained to tell test points from training points.
  2. Conformal prediction and distribution shiftBlack-box shift estimationUnder label shift the class weights q(y)/p(y)q(y)/p(y) solve a k×kk\times k linear system built from any fixed classifier's confusion matrix, which must be invertible and need not be accurate or calibrated.The correction is importance-weighted training with negative estimated weights clipped to zero, best applied only when a shift test rejects, since reweighting without a shift only adds variance.
  3. Cost-sensitive classification and calibrationThe Foundations of Cost-Sensitive LearningA change in base rates multiplies the old posterior by the prior ratio and renormalizes, which in logits adds logit⁡b′−logit⁡b\operatorname{logit}b'-\operatorname{logit}b and uses nothing about the learner.
  4. Cost-sensitive classification and calibrationLocal case-control samplingKeeping examples with a label-dependent probability shifts the log-odds by the log acceptance ratio, which an offset undoes exactly when the model is correct; local case-control keeps each point with probability equal to how surprising its label is under a pilot model.
  5. Missing and selective labelsLearning from positive and unlabeled dataWhen positives are labeled completely at random with probability cc, a labeled-versus-unlabeled classifier equals cc times the true posterior, so it already ranks correctly and dividing by cc calibrates it.
  6. Missing and selective labelsDoubly robust policy evaluationDoubly robust off-policy evaluation importance-weights only the reward model's residual, so its bias is the product of the reward-model and propensity errors and the variance from large inverse-propensity weights scales with the squared residual.Minimizing an importance-weighted loss over many policies picks ones that look good by luck, so the learner clips the weights and penalizes each policy by the spread of its own weighted losses, which keeps it on policies the logged data can evaluate.
  7. Missing and selective labelsThe selective labels problemWhere past decisions hide outcomes and unmeasured confounding rules out any ratio, contraction evaluates a model on the cases released by the most lenient decision-maker, which hold a label for every case a stricter policy would release.

Related threads

Separating probability estimates from decision thresholds

Train the model to output a calibrated probability, and set the decision threshold afterwards from costs and base rates. Rebalancing, logit adjustment and per-transaction thresholds all reduce to moving that threshold, and most errors in a scoring system come from mixing the two steps.

  1. Cost-sensitive classification and calibrationThe Foundations of Cost-Sensitive LearningPredict positive when p≥p∗p\ge p^*, where p∗p^* depends only on the ratio of the two regrets, so the whole cost matrix reduces to one number.In Elkan's card example the threshold for refusing a transaction falls from 0.5 for a zero amount to about 3.8% at 1,000 dollars, a per-example threshold that no single rebalancing of the training set can produce.Thinning the negatives is a base-rate change, so its whole effect on a probability model is a shift of the threshold, and a model trained with negatives kept at rate rr recovers population probabilities when its odds are multiplied by rr.EM on a day's scores estimates a moved fraud rate long before chargebacks arrive, but at a prevalence of a fraction of a percent any calibration error on legitimate traffic corrupts the estimate, so calibration comes first.
  2. Cost-sensitive classification and calibrationLogit adjustmentThe balanced-error Bayes rule subtracts log⁡πy\log\pi_y from each logit, which in the binary case is Elkan's threshold with a false negative costing (1−π)/π(1-\pi)/\pi times a false positive.A model trained with the logit-adjusted loss outputs balanced log-odds, so a downstream rule that reads its score as a probability must add back the logit of the prior it wants.
  3. Cost-sensitive classification and calibrationTemperature scaling and calibrationDividing the logits by one temperature fitted on held-out data restores calibration without changing a single prediction, and on vision benchmarks it mostly beats richer calibrators.A temperature cannot fix a base-rate error, which is an intercept on the logit, and for rare positives calibration has to be checked among the top scores around p∗p^*, where a model with doubled odds predicts 22% against an observed 12.7%.
  4. Conformal prediction and distribution shiftSplit conformal predictionConformal bands get validity from exchangeability alone and width from the model, so any score gets honest coverage and a better model gets a narrower band; the guarantee is marginal and can hide undercoverage of a small high-risk segment.

Related threads

Elsewhere on this site

  • The Price of Changing Someone’s MindIn control the same split is the separation principle, estimate the state and act on the estimate, which this page shows failing once a second player reads your actions in what they observe

Feedback between a model and its data

A deployed model decides which outcomes get observed and provokes responses from the people it scores, so the next training set depends on the current model. Retraining then settles at a stable point that need not be the best model, and whoever adapts more slowly ends up leading. These papers do not cover agents who have private costs or who forecast what the learner will do next.

  1. Strategic and performative predictionPerformative predictionRetraining follows only the part of the gradient with the data held fixed, so when the model's effect on its own data outweighs the curvature, retraining can converge to the worst model.Retraining settles when strong convexity beats how far the data move times how strongly the gradient reacts, so regularization stabilizes the loop and a hard-margin or thresholded rule can cycle at any sensitivity.
  2. Strategic and performative predictionWho leads and who follows in strategic classification?The side that updates more slowly commits and so leads: a fraud team that retrains daily against fraud rings that change tactics weekly makes the rings the leaders, and label delay means a fast update reacts to outdated behavior.
  3. Strategic and performative predictionStrategic classificationUnder a separable gaming cost each feature's weight is its cost of manipulation whatever its predictive value, so a fraud model should lean on signals that are expensive to fake, with the data setting only the threshold.
  4. Missing and selective labelsThe selective labels problemBecause the judge's decision decides whose outcome is observed, evaluating on released cases alone or imputing the rest overstates the model's performance, and a model trained on them can learn that a screened-in group is safe.
  5. Missing and selective labelsDoubly robust policy evaluationOff-policy estimates divide by the logging policy's propensities, so those should be logged and some exploration kept, or the actions a new policy would take are never observed.
  6. Learning from human preferencesReward model overoptimizationOptimizing against a learned reward raises gold score up to a KL that small runs predict and then lowers it, so early stopping or a KL target has a real effect on the gold score.
  7. Conformal prediction and distribution shiftAdaptive conformal inferenceUnder drift, adaptive conformal inference raises the nominal coverage after each miss and lowers it after each hit, which drives the long-run miss rate to α\alpha on any sequence; the paper leaves open the case of labels that arrive weeks late, as chargebacks do.

Related threads

Elsewhere on this site

  • Noise-State Game ExplorerThe noisestate explorer's games keep the full loop that these papers leave out, in which actions change what others observe and believe and players forecast each other's forecasts
  • The Price of Changing Someone’s MindThe information wedge page values an action by how it changes what the other side believes, which matters once agents forecast the learner
  • The Noise-State, ExplainedExplains how the dissertation handles beliefs about beliefs, which agents who forecast the learner would bring in

Fixed scales for stable training

Most training-stability fixes hold the size of some quantity fixed, the update per layer, the attention logits or the output normalizer, so that one learning rate keeps working as width and run length change. When a run diverges or a tuned learning rate stops transferring, look for a quantity whose size drifts with scale.

  1. Lazy training and feature learningMaximal update parametrizationBecause μ\muP keeps every layer learning at the same rate as width grows, hyperparameters tuned on a 40M proxy beat the published 6.7B GPT-3 at 7% of its pretraining cost.
  2. NanoGPT speedrunMuonMeasuring a weight change by how far it moves a unit-RMS input gives the update −ηdout/din UV⊤-\eta\sqrt{d_{out}/d_{in}}\,UV^\top, which moves every output by exactly η\eta whatever the width, which is the μ\muP property, here derived from a choice of norm.
  3. NanoGPT speedrunRefinements to MuonScaling each Muon update to RMS 0.2, the typical RMS of AdamW's update in practice, lets the learning rate and weight decay tuned for AdamW serve Muon unchanged.Muon's steps never shrink as gradients fade, so without weight decay the weight and layer-output RMS drift out of the range where bf16 is precise.Orthogonalized updates have flat singular values but uneven row norms, so NorMuon divides each neuron's row by its running RMS after Newton-Schulz and then restores AdamW's overall update size.
  4. NanoGPT speedrunConvergence theory for MuonWith decay coupled to the learning rate, weight decay sets the radius b/λb/\lambda the weights end up in, and the learning rate sets only how fast they get there.
  5. NanoGPT speedrunSmall-scale proxies for large-scale Transformer training instabilitiesIn the authors' explanation, attention logits are the only quantity quadratic in the parameter scale, so they blow up first as weights grow; normalizing queries and keys per head removes the failure and lets a 1.2B model train at learning rate 0.3.Cross-entropy ignores a constant added to every output logit, so the mean logit drifts, and z-loss 10−4(log⁡Z)210^{-4}(\log Z)^2 holds log⁡Z\log Z near zero.At 4.8B parameters the gradient RMS falls to AdamW's default ϵ=10−8\epsilon=10^{-8}, the update shrinks below its intended size, and lowering ϵ\epsilon to 10−1510^{-15} improves the loss.
  6. NanoGPT speedrunLearning-rate schedules and batch sizeIn the convex bound the best learning rate is a problem constant times a function of the schedule's shape, so ratios transfer, and the bound reproduces the 1/T1/\sqrt T scaling and the factor of about 2 between cosine and warmup-stable-decay schedules found in LLM sweeps.

Related threads