Reinforcement learning for code

Context. In code a reward can be computed by running the program on tests. Earlier RL for code used that signal for single attempts, while the loops used in practice (write code, run it, read the error, repair) were built by prompting, and at an equal number of samples they rarely beat drawing independent solutions. Real software adds a second problem. Most repositories cannot be built and run cheaply, so there are no tests to compute a reward from. Gehring et al. (2025), with RLEFreinforcement learning with execution feedback: RL through multi-turn episodes in which the model sees the results of running its code on example tests before trying again, train Llama 3 models with RL inside the repair loop itself, on competitive programming. Wei et al. (2025), with SWE-RLthe method's name, for RL on software evolution data: the record of code changes, issues and pull requests in public repositories, train on real GitHub issues and reward a generated patch by its textual similarity to the patch the developers merged, with no execution at all. Together they show that a model learns to use feedback when it is trained in the loop where the feedback arrives, and that the history of real repositories supplies a reward informative enough for RL where nothing can be run.

Both papers rest on two design choices. The reward comes from something the model cannot see or copy, held-out tests in RLEF and the developer's own patch in SWE-RL, and the training episode has the shape of the deployment loop. With these, an RLEF-trained 70B model solves 40% of the CodeContestscompetitive programming problems with public and hidden tests; 13,328 training problems (RLEF drops 669 that lack public or private tests), 117 validation and 165 test problems test problems with at most three attempts, more than earlier GPT-4 pipelines reached with up to 100 samples, and its repairs depend on what the feedback says. SWE-RL's 70B model resolves 41.0% of the issues in SWE-bench Verified500 real GitHub issues from Python repositories, checked by humans to be solvable, each graded by running the repository's tests on the submitted patch, ahead of a supervised model trained on the same pull requestsproposed code changes submitted for review and merging into a repository, with their discussion and linked issues, and it matches or beats its base model on every out-of-domain benchmark reported, where the supervised model falls below the base on most.

Setting. An RLEF episode is a dialog: the problem statement, a code answer, the result of running it on the problem's public teststhe example inputs and outputs given with a problem, used for feedback during the episode (failed inputs with expected and actual output, an exception trace, a timeout or a memory error), another answer, and so on. It ends when the public tests pass or after three answersthe turn limit used in training and in the main evaluations, and the final answer is then run on the private testswhy: the model never sees them; graded on the public tests alone, it could copy the expected outputs shown in the feedback into its next answer. With $\pi$the policy being trained, the language model writing each answer, its starting model $\rho$the initial Llama 3 Instruct model, also the reference of the KL penalty and $c_t$the dialog before answer $a_t$: problem statement, earlier answers and their feedback, each answer $a_t$ receives $$R_t=r_t-\beta\log\frac{\pi(a_t|c_t)}{\rho(a_t|c_t)},$$ where the task reward $r_t$$+1$ at the end of an episode whose final answer passes all public and private tests, $-1$ if any test fails, $-0.2$ for an answer with no valid code, otherwise 0, and $\beta$the weight of the KL penalty, 0.05 weights a KL penaltythe log ratio $\log(\pi/\rho)$ subtracted from the reward, whose expectation is the Kullback-Leibler divergence of the trained model from its start. Training uses PPOproximal policy optimization: policy-gradient RL that clips each update so the policy moves only a little, with a learned value function as baseline on CodeContests, starting from the Instruct versions of Llama 3.0 8B and Llama 3.1 8B and 70B. Results are $n$@$k$the expected fraction of problems solved when $n$ final answers, chosen from $k$ generated answers, are submitted; each turn of a dialog counts as one of the $k$ solve rates. An SWE-RL prompt holds an issue, the full contents of the files the merged pull request changed and of some relevant unchanged files, and asks for reasoning followed by search-and-replace edits. A response $o$ receives $$R(o)=\begin{cases}-1&\text{if malformed,}\\ s(\hat p,p^*)&\text{otherwise,}\end{cases}$$ with $s$the ratio of Python's difflib.SequenceMatcher: twice the number of matched elements over the total length of both sequences, a number in $[0,1]$ comparing the predicted patch $\hat p$the patch extracted from the model's edits with the merged patch $p^*$the patch the developers merged for the issue. Llama-3.3-70B-Instruct is trained with GRPOgroup relative policy optimization: PPO-style updates without a value function, where each response's advantage is its reward minus the group mean, divided by the group standard deviation on groups of 16 responses per issue.

Main findings (all empirical). RLEF raises the 1@3one final answer from at most three generated answers, which is one dialog with up to three turns test solve rate, in percent, from 10.5 to 16.0 at 8B and from 27.5 to 40.1 at 70B; the previous best, the GPT-4 agent pipeline AlphaCodiuma hand-designed multi-step pipeline that prompts a model to reflect, generate tests, write code and repair it, scored 29 with 5 answers from 100 samples. At 10@100 the gains are smaller (50.3 to 54.5 at 70B), which the authors read as RL reducing the diversity of outputs. Before RLEF, a feedback dialog gains little or nothing over independent answers at the same budget (the same $k$), for Llama or GPT-4o; after it, the dialog is clearly ahead, and repairs largely fail when the feedback comes from an unrelated problem. Supervised fine-tuning on successful dialogs, or RL on single answers, falls well short. SWE-RL resolves 41.0% of SWE-bench Verified against 36.2% for the supervised baseline; given the right files and one greedy answer, it repairs 34.8% of issues against 29.6% for supervised training and 5.4% for the base model, and its similarity reward beats an exact-match reward (34.8 against 29.0).

Consequences and limits. For a coding agent the lesson is to train in the deployment loop, reward on tests the model cannot see, and count every attempt against the budget; feedback helps mainly a model trained to use it. When no tests can be run, similarity to real human fixes is a usable reward, and repository history supplies far more training issues than curated benchmarks. The limits differ by paper. RLEF improves one self-contained program per problem, needs test cases, and its gains shrink at large sample budgets. SWE-RL's reward measures how closely a patch resembles the developer's, so a correct fix written differently gets a low reward, a limitation the authors state; its benchmark score also depends on a pipeline with 500 samples per issue and test-based reranking, and training never involved execution. Neither paper reports error bars on its main benchmark.