Sam Babichenko

Reading notes

In-context learning

Algorithms a transformer runs on the examples in its prompt.

In-context learning of simple function classesGarg et al., 20227 questions Transformers Learn In-Context by Gradient Descentvon Oswald et al., 2023; Ahn et al., 2023; Zhang et al., 20248 questions Transformers as StatisticiansBai et al., 20238 questions In-context learning as implicit Bayesian inferenceXie et al., 2022; Raventós et al., 20238 questions Induction headsOlsson et al., 20227 questions