LLMs Can Learn to Reason Via Off-Policy RL

Context. Policy-gradient methods such as PPOproximal policy optimization: a policy-gradient method that multiplies each token's advantage by its probability ratio, current over sampling policy, and clips the ratio so each update moves the policy only a little and GRPOgroup relative policy optimization: PPO without a value model, in which every token of an answer gets the answer's reward minus its group's mean reward, divided by the group's standard deviation, as advantage assume that the training data come from the policy being trained, and in LLM post-training they rarely do. Answers are generated by a separate inference enginesoftware that generates text from a model's weights at high throughput, such as vLLM; it computes probabilities with different numerical kernels from the training code whose log-probabilities differ slightly from the trainer's even with identical weights, and in asynchronous pipelines that engine holds weights several updates old. Most fixes make the data look on-policy again: they multiply GRPO's loss by importance ratiosa sample's probability under the policy being trained divided by its probability under the policy that generated it between trainer and inference engine, per token or per answer and usually truncated or masked, or they change the inference engine so that its numbers match the trainer's. Ritter et al. (2026) instead use an objective meant for off-policy data, a KL-regularizedmaximizing expected reward minus a weight times the Kullback-Leibler divergence of the policy from an anchor policy problem whose anchor is the policy that generated the data. Their method, OAPL (optimal advantage-based policy optimization with lagged inference policy), fits that problem by least squares and updates the inference engine's weights only every 50 to about 400 trainer steps.

If the KL penalty is measured against the sampling policy itself, the best policy for the regularized problem is the sampler reweighted by $e^{R/\beta}$, with $R$ the reward and $\beta$ the KL weight, and its log-ratio to the sampler is the reward minus a per-question constant, divided by $\beta$. The constant is computed from an average under the sampler, so a group of the sampler's answers estimates it. Fitting $\beta$ times the trained policy's log-ratio to that target by least squares uses only the sampler's answers and its log-probabilities, so there is no ratio to correct and nothing to clip or truncate. The gradient of the fit is a policy gradient weighted by the remaining gap to the target, which goes to zero once an answer's log-probability has moved far enough. Repeated with a periodic update of the inference engine, this is a sequence of KL-regularized improvement steps, each computed from the samples of one fixed sampler. With the engine updated every 50 trainer steps, OAPL beat GRPO with importance sampling on three 2025 math competitions; with about 400 steps between updates it matched DeepCodera 14B code model trained from DeepSeek-R1-Distill-Qwen-14B with GRPO plus extra heuristics such as a raised upper clip and overlong filtering (Luo et al., 2025), a GRPO-trained code model, on LiveCodeBencha benchmark of recent competitive-programming problems scored by hidden tests with about a third as many samples.

Setting. For a question $q$, the inference engine's policy $\pi_{\rm old}$the policy whose weights the inference engine holds: it samples the data and is the anchor of the KL penalty (the paper writes $\pi_{\rm vllm}$); $\pi_\theta$ is the policy being trained samples $G$the number of answers sampled per question, 8 in the experiments answers $o_1,\dots,o_G$ with rewards $R_i=R(q,o_i)$, 1 for a correct answer and 0 otherwise. Write $\ell_i$$\ell_i=\log\pi_\theta(o_i\mid q)-\log\pi_{\rm old}(o_i\mid q)$, the log-ratio of the whole answer: the sum of its token log-ratios, with no division by length for the log-ratio of answer $o_i$, $\ell_i=\log\pi_\theta(o_i\mid q)-\log\pi_{\rm old}(o_i\mid q)$. OAPL targets the KL-regularized problem with $\pi_{\rm old}$ as anchor and KL weight $\beta$the weight of the KL penalty; larger values keep the solution closer to $\pi_{\rm old}$, with no KL term to a fixed reference model:$$\begin{aligned}\max_\pi\ &\mathbb E_{o\sim\pi(\cdot\mid q)}R(q,o)\\&-\beta\,\mathrm{KL}\big(\pi(\cdot\mid q)\,\|\,\pi_{\rm old}(\cdot\mid q)\big).\end{aligned}$$Its solution $\pi^\star$$\pi^\star(o\mid q)=\pi_{\rm old}(o\mid q)\,e^{(R(q,o)-V^\star(q))/\beta}$, the anchor reweighted by $e^{R/\beta}$ and normalized is the anchor reweighted by $e^{R/\beta}$, normalized by the optimal value $V^\star(q)$$V^\star(q)=\beta\log\mathbb E_{o\sim\pi_{\rm old}(\cdot\mid q)}e^{R(q,o)/\beta}$, the largest value the regularized objective can reach for question $q$, which the group estimates by$$\hat V(q)=\beta\log\Big(\frac1G\sum_{i=1}^Ge^{R_i/\beta}\Big).$$With $A_i$$A_i=R_i-\hat V(q)$, the estimated optimal advantage of $o_i$ $=R_i-\hat V(q)$, OAPL minimizes, over the questions in its buffer,$$\sum_q\sum_{i=1}^G\big(\beta\,\ell_i-A_i\big)^2.$$The procedure: start with $\pi_\theta=\pi_{\rm old}$; the inference engine generates groups into a buffer while the trainer takes gradient steps on this loss using data from the buffer; every $L$the number of trainer updates between synchronizations of the inference engine: 50 or 100 in the math experiments, 418 in the code experiment trainer steps the engine is synchronized to $\pi_\theta$ and the buffer is emptied, so that each $\hat V$ and each loss term use samples from a single $\pi_{\rm old}$. In the experiments the $\beta$ inside $\hat V$ and the one multiplying $\ell_i$ are set separately, $\beta_1=1$ and $\beta_2=10^{-3}$the values of $\beta$ in $\hat V$ and in the loss, chosen from $\beta_1\in\{1,5\}$ and $\beta_2\in\{10^{-2},10^{-3}\}$ in the math runs and reused for code.

Main findings (empirical apart from the optimality argument). (a) With the exact $V^\star$, the policy $\pi^\star$ makes every term of the loss zero, so it minimizes the loss whatever distribution the answers were drawn from. (b) On competition math with Qwen3-4B-Thinking-2507 and three runs per method, OAPL with $L=50$ is ahead of GRPO with an importance-sampling correction and at most one step of lag, on AIME 2025, HMMT 2025 and BRUMO 20252025 high-school competitions with short exact answers: the American Invitational Mathematics Examination, HMMT (February and November) and the Brown University Math Olympiad at pass@kthe probability that at least one of $k$ sampled answers is correct, estimated from at least $k$ samples per problem with the unbiased estimator of Chen et al. (2021) for $k=1$ to 8, and still trains stably with $L=100$. (c) GRPO's entropy collapses during training and OAPL's does not, and after training both raise pass@k above the base model for every $k$ up to 256, OAPL more so. (d) On code, starting from DeepSeek-R1-Distill-Qwen-14B with two rounds of training on fixed data and a lag the paper puts at about 400 updates, OAPL matches or slightly beats DeepCoder on LiveCodeBench from pass@1 to pass@10, with about 200,000 generated answers against about 650,000.

Consequences. When the mismatch between trainer and inference engine, or the lag of an asynchronous pipeline, is what destabilizes GRPO, an alternative to adding importance ratios is to change the loss so that the sampler is the anchor. The loss is one scalar regression per answer: its summed log-ratio, from the trainer's log-probabilities and the ones the inference engine returned while sampling, regressed onto $R_i-\hat V(q)$; there is no clip, no importance ratio and no copy of the previous trainer policy. The synchronization interval $L$ then acts like a step size, since the anchor moves only at a synchronization and in between the policy is pulled toward a fixed regularized target. The paper's pass@k result also bears on whether RL with verifiable rewards only sharpens what the starting model can already do; in this setup it widened the range of problems solved within $k$ samples (Step 5).

Limits. The math comparison runs OAPL at a lag of 50 steps against GRPO at a lag of one, so it shows that OAPL tolerates the larger lag but not how GRPO with importance sampling would do at the same lag; the abstract's "100x more off-policy than prior approaches" compares the lag OAPL tolerates with the lags used in earlier work, without running those methods at the same lag. OAPL's $\beta$'s were chosen from a small search, while GRPO's settings are reported without one. Checkpoints are chosen by accuracy on the test benchmarks (best average pass@1 in math, best of four in code). Each task uses one starting model, both already reasoning models, and the code result is a single training run compared with a DeepCoder whose reported score could not be reproduced.