Reading notes

Estimators that fit noisy training data exactly and still predict well.

1. Stein's paradox and the James-Stein estimatorStein, 1956; James and Stein, 1961; Efron and Morris, 1973
2. Double descentBelkin et al., 2019; Belkin et al., 2020
3. Benign Overfitting in Linear RegressionBartlett et al., 2020; Tsigler and Bartlett, 2023
4. Harmless interpolation of noisy data in regressionMuthukumar et al., 2020
5. Surprises in High-Dimensional Ridgeless Least Squares InterpolationHastie et al., 2022
6. Dimension free ridge regressionCheng and Montanari, 2024
7. Uniform convergence of interpolatorsKoehler et al., 2021; Zhou et al., 2021
8. Kernel interpolationLiang and Rakhlin, 2020; Belkin et al., 2019
9. Benign, Tempered, or CatastrophicMallinar et al., 2022
10. The implicit bias of gradient descentSoudry et al., 2018

Uniform convergence, margin and PAC-Bayes bounds for deep networks.

1. Understanding deep learning requires rethinking generalizationZhang et al., 2017
2. Spectrally normalized margin boundsBartlett et al., 2017
3. Limits of uniform convergenceNagarajan and Kolter, 2019
4. Nonvacuous PAC-Bayes boundsDziugaite and Roy, 2017; Lotfi et al., 2022

Infinite-width limits of neural networks in which the features stay fixed or move.

1. The neural tangent kernelJacot et al., 2018; Chizat et al., 2019
2. Mean-field limits of two-layer networksMei et al., 2018
3. Maximal update parametrizationYang and Hu, 2021

How many samples SGD needs to find a hidden direction in high dimension.

1. The information exponentBen Arous et al., 2021; Damian et al., 2023
2. Feature learning from one gradient stepBa et al., 2022; Damian et al., 2022
3. The merged-staircase propertyAbbe et al., 2022
4. Learning sparse parities with SGDBarak et al., 2022

Power-law learning curves and compute-optimal model size.

1. Spectrum-dependent learning curvesBordelon et al., 2020; Maloney et al., 2022
2. Explaining Neural Scaling LawsBahri et al., 2021
3. Chinchilla scalingHoffmann et al., 2022
4. 4+3 Phases of Compute-Optimal Neural Scaling LawsLin et al., 2024; Paquette et al., 2024

Algorithms a transformer runs on the examples in its prompt.

1. In-context learning of simple function classesGarg et al., 2022
2. Transformers Learn In-Context by Gradient Descentvon Oswald et al., 2023; Ahn et al., 2023; Zhang et al., 2024
3. Transformers as StatisticiansBai et al., 2023
4. In-context learning as implicit Bayesian inferenceXie et al., 2022; Raventós et al., 2023
5. Induction headsOlsson et al., 2022

Networks that represent more sparse features than they have dimensions, and recovering those features.

1. A Mathematical Framework for Transformer CircuitsElhage et al., 2021
2. Toy Models of SuperpositionElhage et al., 2022
3. Decoding by Linear ProgrammingCandès and Tao, 2005
4. Sparse dictionary learningBricken et al., 2023

Theory of score-based generative models.

1. Score-based diffusionSong et al., 2021
2. Sampling is as easy as learning the scoreChen et al., 2023
3. Stochastic localizationMontanari, 2023
4. Generalization in diffusion modelsKadkhodaie et al., 2024

Changes that reduced GPT-2 small training time from 45 minutes to under a minute, and the papers behind them.

1. The NanoGPT speedrunJordan et al., 2024–2026
2. MuonJordan et al., 2024; Bernstein and Newhouse, 2024; Bernstein, 2025; Amsel et al., 2025
3. Convergence theory for MuonKovalev, 2025; Chen et al., 2025
4. Refinements to MuonLiu et al., 2025; Li et al., 2025; Chen et al., 2025; Srivastava, 2025
5. Learning-rate schedules and batch sizeSchaipp et al., 2025; McCandlish et al., 2018
6. Small-scale proxies for large-scale Transformer training instabilitiesWortsman et al., 2023
7. FlexAttentionDong et al., 2024; records #12-#13
8. Cross-layer connectionsZhou et al., 2024; Xiao et al., 2025
9. Conditional Memory via Scalable LookupCheng et al., 2026

Reward models, KL-regularized fine-tuning and direct preference optimization.

1. InstructGPTOuyang et al., 2022
2. Direct preference optimizationKorbak et al., 2022; Rafailov et al., 2023; Azar et al., 2024
3. Reward model overoptimizationGao et al., 2023

Positive-unlabeled learning, selective labels and off-policy evaluation.

1. Learning from positive and unlabeled dataElkan and Noto, 2008; Kiryo et al., 2017
2. The selective labels problemLakkaraju et al., 2017
3. Doubly robust policy evaluationDudík et al., 2011; Swaminathan and Joachims, 2015

Decision thresholds from costs and base rates, and calibrated scores.

1. The Foundations of Cost-Sensitive LearningElkan, 2001; Saerens et al., 2002
2. Local case-control samplingFithian and Hastie, 2014
3. Temperature scaling and calibrationGuo et al., 2017; Niculescu-Mizil and Caruana, 2005
4. Logit adjustmentMenon et al., 2021

Distribution-free prediction sets, and corrections when test data differ from training data.

1. Split conformal predictionLei et al., 2018; Romano et al., 2019
2. Conformal Prediction Under Covariate ShiftTibshirani et al., 2019
3. Adaptive conformal inferenceGibbs and Candès, 2021
4. Black-box shift estimationLipton et al., 2018

Prediction when the people or data being predicted respond to the model.

1. Strategic classificationHardt et al., 2016; Brückner et al., 2012
2. Performative predictionPerdomo et al., 2020; Miller et al., 2021
3. Who leads and who follows in strategic classification?Zrnic et al., 2021

Published transaction and banking models, temporal graphs and fraud detection.

1. Transaction foundation models in industryEight published transaction and banking foundation models.
2. Pretraining objectives and encodingsWhat CPC, replaced-event detection and temporal contrast learn, and what they miss.
3. Temporal graphs and real-time servingMemory, neighbors and honest evaluation on timestamped graphs; serving in real time.
4. Relational deep learning and graph foundation modelsGraphs built from relational tables, graph pretraining, and graph foundation models.
5. Fraud graphs and fraud labelsFraud GNNs against tree baselines, edge-level crime patterns, and the trouble with labels.