LLMs are greedy agents

Context. Language models are being used as agents on the hypothesis that their knowledge and step-by-step reasoning let them act well after little interaction. On the simplest test of acting under uncertainty, the multi-armed bandita repeated choice among $k$ options (arms), each giving a random reward from its own unknown distribution; the agent must try arms to learn which is best while collecting reward, they explore too little, and in games they can often describe the right move without making it, a failure called the knowing-doing gapthe gap between what a model can state or compute about a task and what it actually does when acting. Schmied et al. take Gemma2 models at 2B, 9B and 27B, measure why they fall short on text bandits, and then fine-tune the 2B and 9B models with RL on their own chain of thoughtreasoning text the model writes before its answer and the environment's rewards, on bandits and (2B only) Tic-tac-toe. Their finding is that pretrained models choose actions from the statistics of their context, the small one repeating the most frequent action and the larger ones committing early to the best action seen, and that RL on environment rewards makes them explore more but not as well as a bandit algorithm, so information and exploration still have to be supplied or rewarded. It carries the in-context-learning question, what a model learns from its context, over to choosing actions.

Exploration is the missing part. With reasoning, the models try 40 to 65% of 10 arms in 50 steps and stop trying new ones after about 10 steps; without it they try a quarter. The 2B model mostly repeats whatever action fills its context, and the 27B model, which does not, computes the textbook exploration rule, UCB (a bonus that shrinks with each pull), correctly in 87% of steps and still acts greedily in most of them. RL fine-tuning lowers regretcumulative regret: the expected reward lost against always pulling the best arm, summed over steps, widens the set of arms tried and halves the copying of the most frequent action when it has been repeated only a few times, and the largest further gains come from giving the model information (trying every arm once) or rewarding exploration directly (+1 for each new arm).

Setting. The bandits come from BanditBencha suite of bandit tasks posed in text to language models, from Nie et al.: $k\in\{5,10,20\}$ arms described as colored buttons (or as numbers), Gaussian rewards with means $\mu$an arm's mean reward, drawn uniformly from $[0,1]$ for each bandit and noise $\sigma$the reward standard deviation: 0.1, 1 or 3 (low, medium or high noise), and a horizon of $T$the number of steps per episode, 50 steps. At step $t$ the model reads instructions and the recent history as lines such as "Step=3 Action=red Reward=0.5", writes up to $G$the token budget per step for reasoning plus the action, 256 by default tokens of reasoning followed by an action, and the action $a_t$the arm chosen at step $t$: the last match of the pattern ACTION=X in the generated text, or a random arm if none is found is executed. With $\mathcal A$the set of arms and $N_t(a)$the number of times arm $a$ was chosen up to step $t$, exploration is measured by action coverage$\mathrm{cov}_t=|\{a\in\mathcal A:N_t(a)\gt0\}|/|\mathcal A|$, the fraction of arms tried at least once by step $t$ and performance by cumulative regret. The references are UCBupper confidence bound: pull each arm once, then always the arm with the highest mean reward plus a bonus that shrinks as the arm is pulled more, which the paper treats as the upper bound on performance, and a uniformly random agent as the lower bound. RLFTreinforcement learning fine-tuning, the paper's name for RL on the model's own reasoning and actions with rewards from the environment trains the policy $\pi_\theta$the language model as a policy: given the context, a distribution over generated text on its own reasoning and actions with PPOproximal policy optimization: policy-gradient RL that clips each update so the policy moves only a little's clipped objective and a KL penalty to the starting model $\pi_{\rm ref}$: $$\begin{aligned}\max_\theta\ &\mathbb E\Big[\underbrace{\min\big(\rho\hat A,\ \mathrm{clip}_\epsilon(\rho)\hat A\big)}_{\text{clipped surrogate}}\Big]\\&-\underbrace{\beta\,\mathrm{KL}(\pi_\theta\|\pi_{\rm ref})}_{\text{KL penalty}},\end{aligned}$$ where $\rho$the ratio $\pi_\theta(z|c)/\pi_{\rm old}(z|c)$ of the new to the sampling policy's probability of the generated text $z$ given the context $c$ is the probability ratio, $\hat A$rewards-to-go minus a Monte Carlo baseline for bandits; from a learned value head with generalized advantage estimation for Tic-tac-toe the advantage, the clipping range $\epsilon=0.2$ and the KL weight $\beta=0.05$. The reward adds a penalty of $-5$ when no valid action can be extracted. Training runs 30k updates over a pool of 512 bandits, 16 per rollout.

Main findings (all empirical). On 10-arm button bandits the 2B model covers 40% of arms and the 9B and 27B models 65%, all three about 25% without reasoning, and on 20 arms no model passes 45%. In histories where one action is repeated, the 2B model picks the most frequent action 96% of the time and the 27B model 14%. Asked to compute UCB's values first, the 27B model gets UCB's choice right in about 87% of steps, but in steps with a correct computation it takes the greedy action 64% of the time (the paper's introduction) and UCB's action 24% (my computation from its confusion matrix, 21.6/90.3). RLFT on 2B lowers regret for 5, 10 and 20 arms at every noise level, raises action coverage by 12%, and cuts frequent-action choices from 70% to 35% at low repetition. An exploration bonus during RLFT raises coverage from 50% to 70%. In Tic-tac-toe, RLFT lifts the 2B model's average return against a random opponent from 0.15 to 0.75 and against MCTSMonte Carlo tree search, a game-playing search algorithm, here the strongest opponent from −0.95 to about 0. Fine-tuning on UCB's own trajectories comes close to UCB's regret.

Consequences and limits. An LLM agent that must gather information does better when it already has an observation of every action (trying each arm once) or is told which moves are legal, when it can reason with enough tokens, and, during RL fine-tuning, when exploration is rewarded directly; demonstrations from an expert, where they exist, work better than RL from the model's own attempts. RL on the model's own reasoning narrows the gap between what it computes and what it does, but it does not turn the model into a bandit algorithm. The study is small: one model family up to 27B, RL only at 2B and 9B, horizons of 50 steps (too short for 20 arms), toy environments, the knowing-doing measurement only for the 27B model, and three seeds per result.