Laminar: asynchronous RL at scale

Policy and staleness
$\theta_j;$ $\pi_{\theta_j}$	weights the trainer publishes after j steps; the policy they define
$\pi_\theta(y)$	probability of trajectory y under the current weights
$J(\theta)$	expected reward of the policy with weights θ
$y;$ $y_t$	a trajectory; its token t
$A(y)$	advantage of a trajectory, GRPO's standardized group reward
$\rho_t$	token probability ratio, trained over generating policy
$n-j$	inherent staleness: training steps completed while a trajectory was generated
$k$	staleness bound of other asynchronous systems
$\epsilon_{\rm low},$ $\epsilon_{\rm high}$	lower and upper clipping thresholds of the ratio
Repacking
$C_{\rm used}$	fraction of a rollout's KV cache in use
$C_{\max}$	utilization of a full cache
$B$	largest decode batch that runs at small-batch latency
Broadcast
$p$	relays in the chain, master included
$W;$ $c$	size of the weights in bytes; number of chunks
$T_{\rm s};$ $T_{\rm b}$	startup time per message; transfer time per byte
$T^*(p)$	broadcast time at the best number of chunks
Scaling
$X;$ $G$	throughput; GPU count, at the largest and smallest cluster
Constants
0.2; 0.28	$\epsilon_{\rm low}$ and $\epsilon_{\rm high}$ in the GRPO runs (AReaL: 0.2 and 0.2)
512 × 16	prompts per step × responses per prompt
99%	$C_{\max}$ , the repacking threshold (Step 3)
5 s	interval of the repacking check (Step 3)
4	largest staleness observed in any experiment (Step 5)
2,048; 512	training mini-batch of the asynchronous runs; of verl (convergence runs)

Context. One step of RL post-training has two phases: rolloutscopies of the model, each on its own GPUs, that generate the trajectories (responses, possibly with tool calls) used for training generate trajectories, and the trainer updates the weights on them. Generation dominates: in math reasoning it took up to 83.1% of a step in the runs of DAPOan open RL recipe for math reasoning whose training runs supplied this time breakdown, and lengths are so skewed that the 99th percentile can be ten times the median, so most GPUs sit idle while the last long trajectories finish. Asynchronous systems overlap the two phases by letting rollouts generate with weights up to $k$ the staleness bound: the largest number of training steps by which a rollout's weights may lag the trainer's training steps old, but they keep a global synchronization: at fixed points in the schedule all rollouts receive the new weights together, so all of them still wait for the slowest trajectory, and $k$ trades throughput against staleness. Systems that instead interrupt all generation to load new weights must rebuild each interrupted trajectory's KV cachethe stored attention keys and values of a sequence's earlier tokens, kept so that each new token needs one forward step; rebuilding it means re-running the whole prefix and produce trajectories generated by several weight versions. Sheng et al. make the single trajectory the unit of asynchrony: each rollout fetches the newest weights whenever it finishes its own batch, so every trajectory comes from one weight version and its staleness is set by its own generation time instead of a global schedule. It is the systems counterpart of the finding, in the study of RL scaling curves (Khatri et al., 2026), that overlapping generation with training speeds up training without lowering the final pass rate, although the pipeline in that study loaded new weights in the middle of trajectories, which Laminar avoids.

Relay workers, one CPU process per rollout machine, hold the newest weights in host memory, so the trainer hands its weights to one master relay and resumes training, and any rollout loads them whenever it is ready (Step 2). Repacking moves the unfinished trajectories of rollouts stuck on their long tail onto a few rollouts with the same weights, so that the freed ones refresh and generate fresh data (Step 3). On up to 1,024 GPUs this gives on average 2.56 times the training throughput of synchronous training and 1.39 times that of the fastest asynchronous baseline, and up to 5.48 times in the best case. In wall-clock time the training reward of 7B and 32B models rises 1.77 and 1.59 times faster than with the best baseline, which for convergence was synchronous training. Laminar sets no staleness bound: staleness stayed typically below 3 steps, and the largest value observed was 4.

Setting. The trainer runs GRPOgroup relative policy optimization: PPO-style clipped updates without a value model, where each response's advantage is its reward standardized within the group of responses to the same prompt with clip-higherclipping each token's probability ratio to $[1-\epsilon_{\rm low},1+\epsilon_{\rm high}]$ with $\epsilon_{\rm high}\gt\epsilon_{\rm low}$ , here 0.28 and 0.2, as in DAPO and a rule-based reward on DAPO-Math-17ka public set of about 17,000 math problems with integer answers, released with DAPO, with prompts of up to 2K tokens and responses of up to 16K, 512 prompts with 16 responses each per step, and 16 mini-batch updates per step, after which the trainer publishes new weights; a second task adds multi-turn code execution in a sandbox, with up to 8 tool calls. The models are Qwen2.5 at 7B, 32B and 72B, on up to 128 machines of 8 H800an NVIDIA data-center GPU with 80 GB of memory, a variant of the H100 GPUs. Throughput is the number of prompt and response tokens in a step divided by the time between two weight publications. Write $\theta_j$ the weights the trainer publishes after $j$ steps; a rollout that uses them generates with policy $\pi_{\theta_j}$ for the weights after $j$ steps. A trajectory generated with $\theta_j$ and finished when the trainer holds $\theta_n$ has inherent staleness $n-j$ : the number of training steps the trainer completed while the trajectory was being generated $n-j$ .

What staleness does to the gradient. The policy gradient, the gradient of the expected reward $J(\theta)$ , is an expectation under the current policy $\pi_\theta$ the model with weights $\theta$ as a distribution over trajectories; $\pi_\theta(y)$ is the probability of trajectory $y$ (the prompt is left out of the notation), while the data come from $\pi_{\theta_j}$ . Reweighting by the ratio of the two probabilities makes the expectations agree: $\begin{aligned}\nabla J(\theta)&=\mathbb E_{y\sim\pi_\theta}\big[A(y)\,\nabla\log\pi_\theta(y)\big]\\&=\mathbb E_{y\sim\pi_{\theta_j}}\Big[\underbrace{\tfrac{\pi_\theta(y)}{\pi_{\theta_j}(y)}}_{\text{correction}}\\&\qquad\times A(y)\,\nabla\log\pi_\theta(y)\Big],\end{aligned}$ with $A(y)$ the advantage of trajectory $y$ : its reward minus a baseline; in GRPO, the reward minus the group's mean, divided by the group's standard deviation. Dropping the correction gives an estimate that is exact only at $\theta=\theta_j$ , and keeping it means multiplying the ratios of all the trajectory's tokens, a weight whose spread grows with $n-j$ . PPO-style losses, GRPO among them, use instead each token's own ratio $\rho_t$ $\pi_\theta(y_t\mid y_{\lt t})/\pi_{\theta_j}(y_t\mid y_{\lt t})$ , the probability of token $t$ given the tokens before it, under the trained policy over the generating one, $\rho_t=\pi_\theta(y_t\mid y_{\lt t})/\pi_{\theta_j}(y_t\mid y_{\lt t})$ , clipped. This corrects each token's probability given its prefix and ignores the change in the distribution of the prefix itself, so it is accurate only while the two policies are close. More staleness therefore means more tokens whose ratio has moved past the clipping range in the direction their advantage favors, which then contribute no gradient, and a larger error in the tokens that remain. The paper describes no correction beyond GRPO's clipped ratio and relies on keeping $n-j$ small. All of this assumes that the ratio's denominator is the generating policy's probability, which the paper does not state (Technical notes).

A trajectory was generated with weights 3 steps older than the trainer's. What goes wrong if it is used in a GRPO update whose clipped token ratios are taken against the generating policy?

Main findings (all empirical). On math, Laminar's throughput averages 2.56 times that of verlan open RL training framework that runs generation and training one after the other on the same GPUs; the synchronous baseline (up to 5.49 times), 1.98 and 1.93 times those of one-step-stalea pipeline in which rollouts generate the next batch with the previous step's weights while the trainer trains on the last batch and streaminga one-step-stale pipeline in which the trainer starts on the short trajectories of a batch while the long ones are still being generated pipelines the authors built on verl, and 1.39 times that of AReaLan asynchronous RL system that interrupts generation to load new weights and continues each trajectory with them, with a modified PPO loss for the mixed versions (up to 1.81 times); on the tool-calling task it averages 2.62 times the baselines. The gain grows with the cluster: 1.41 times on average at 64 GPUs or fewer, 3.34 times at the largest scales. The relays cut a rollout's wait for new weights by up to 37% on average, repacking adds 26% to generation throughput, and a killed rollout machine is replaced in about 252 s while training continues.

Consequences and limits. When generation dominates a step and trajectory lengths are skewed, the bottleneck is the global weight update, and more rollout GPUs do not remove it. Letting each rollout refresh on its own schedule removes it and turns staleness into a measured quantity instead of a tuned bound. The convergence comparison also shows that asynchrony helps only while staleness stays small: the other asynchronous systems processed more tokens per second than synchronous training but reached a given reward later. The evidence has limits. Throughput is averaged over five steps from fixed intermediate checkpoints, so it measures the speed of a step at one length distribution and leaves out how speed changes over a run. Convergence is shown by training-reward curves on one math dataset for two models, with no held-out accuracy; the asynchronous runs used training mini-batches of 2,048 against 512 for verl, and AReaL used its own algorithm with a learning rate of $2\times10^{-5}$ against $10^{-6}$ . Two baselines are the authors' reimplementations, and the loss of reward per step of staleness is not measured.

Step 1

A global weight update makes every rollout wait for the slowest trajectory. Decoding is memory-boundlimited by the speed of reading the weights and the KV cache from GPU memory, so adding sequences to a decode batch adds little time: for Qwen2.5-7B and 32B on H800s, a decode batch of fewer than 8 sequences takes nearly as long per step as one of 64, and spreading a rollout over more GPUs with tensor parallelismsplitting each weight matrix across several GPUs so that they compute each layer together shortens a step only a little. A rollout left with a few long trajectories therefore runs at nearly full cost for a small share of the output, and adding rollouts does not shorten the wait for the longest one. One-step-stale pipelines train on the previous full batch while the next is generated, and streaming pipelines train early mini-batches on the short trajectories first, but in both the trainer cannot finish an iteration before the batch's last trajectory arrives, and all rollouts switch weights at once. Interrupting every trajectory at each update removes the wait at the cost of rebuilding the KV cache of every interrupted trajectory and of mixing weight versions within a trajectory, which the authors find slows convergence (Step 5).

Step 2

Relay workers in host memory let the trainer publish weights once and let rollouts fetch them at any time. When rollouts refresh at different moments, a request can arrive while the trainer is in the middle of a step, before its new weights exist. Serving it GPU to GPU needs a spare copy of the previous weights in GPU memory, which training batches and KV caches already fill, or else the trainer stalls until every rollout has pulled. Shared storage is too slow at this size: serializing one 4 GB shard of a 32B model took about 8 s, and moving it over TCP another 10 to 20 s. Laminar runs one relay process in the CPU memory of each rollout machine. The trainer sends its weights by RDMAremote direct memory access: network transfer straight between the memories of two machines without involving their CPUs to a master relay and resumes training; the master reshards them for the rollouts' layout and passes them down a chain of relays in chunks, each relay forwarding one chunk while it receives the next; a rollout loads them from its own machine's relay over PCIethe bus between a machine's CPU memory and its GPUs. For a chain of $p$ the number of relays in the broadcast chain, the master included relays and weights of $W$ the size of the model weights in bytes bytes sent in $c$ the number of chunks the weights are split into for the pipelined broadcast chunks, with startup time $T_{\rm s}$ the fixed latency of one message between neighboring relays, microseconds over RDMA per message and transfer time $T_{\rm b}$ the time to send one byte between neighbors, the inverse of the link bandwidth per byte, the first chunk crosses $p-1$ hops and the other $c-1$ chunks arrive one chunk time apart, so the broadcast takes $(p+c-2)(WT_{\rm b}/c+T_{\rm s})$ , and minimizing over $c$ gives $\begin{aligned}T^*(p)&=\underbrace{WT_{\rm b}}_{\text{one copy}}+\underbrace{(p-2)T_{\rm s}}_{\text{startups}}\\&\quad+\underbrace{2\sqrt{(p-2)WT_{\rm b}T_{\rm s}}}_{\text{pipeline}},\end{aligned}$ which for gigabytes of weights and microsecond startups is close to the time of one copy even for thousands of relays. Measured, a 72B model reaches 127 relays in under 1.6 s, the trainer stalls 0.64 s (32B) and 1.40 s (72B) per update whatever the number of rollouts, and from 64 to 1,024 GPUs a rollout's wait for new weights is up to 37% shorter on average, and 47% in the best case, than with GPU-to-GPU global broadcast. The average stays near the best case because rollouts finish at different times, so few ask for weights before the broadcast reaches their relay.

Why do the relays keep the weights in the CPU memory of the rollout machines instead of sending them GPU to GPU whenever a rollout asks?

Step 3

Repacking moves the long tail onto a few rollouts so that the others can refresh. A rollout that has run out of queued prompts keeps decoding its last few trajectories at almost full cost (Step 1), and it cannot fetch new weights until they finish, so it keeps producing stale data. Laminar detects this from the rollout's KV-cache utilization $C_{\rm used}$ the fraction of the rollout's KV-cache memory in use. It rises to a ceiling $C_{\max}$ the utilization at which the KV cache is full, about 99%, stays there while queued trajectories take the space that finished ones free, and falls once the queue is empty, so the fall marks the tail without a per-task threshold that would have to be profiled. With $B$ the largest decode batch whose step time stays close to that of a small batch, from a roofline model of the GPU: decoding is memory-bound below it and compute-bound above, a rollout becomes a candidate when its utilization is falling and below $C_{\max}$ and it holds fewer than $B$ trajectorieswhy: a falling, unfilled cache means no queued work is left, only the tail; the cap $B$ keeps the merged batch in the range where decoding stays memory-bound, so merging does not slow it. Within each group of rollouts that hold the same weight version, a best-fit bin-packing pass, quadratic in the number of candidates, empties the candidates with the least KV-cache use first, each onto the candidate that will be fullest afterwards without exceeding $C_{\max}$ or $B$ , and the emptied rollouts fetch the newest weights. The check runs every 5 s and right after each weight update. On the 32B model with 64 trainer GPUs and 16 rollouts of 4 GPUs, repacking raises generation throughput by 26% and average KV-cache utilization from 71.6% to 82.2%, takes 0.69 s per repack, and leaves trajectory latency unchanged (average and maximum 290 and 828 s with it, 296 and 826 s without).

Step 4

The throughput gain grows with the size of the cluster. Each system's split of GPUs between trainer and rollouts was tuned to balance the two phases, the global batch was held fixed as GPUs were added, and the staleness bound was 1 for streaming generation and unbounded for AReaL. Laminar's strong-scaling efficiency $(X_{\max}/X_{\min})/(G_{\max}/G_{\min})$ for throughput $X$ and GPU count $G$ at the largest and smallest cluster; 100% means throughput grows in proportion to the GPUs is 53.7% on math (68.2% for the 32B model) and 46.5% on tool calling, against at best 33.6% (AReaL on math, at most 39.2% for one model) and 12.9% (streaming generation on tool calling) for the baselines. At 64 GPUs or fewer the gain averages 1.41 times, because each model needs a minimum number of GPUs for training and for generation, which leaves all systems with nearly the same placement. At the largest scales it averages 3.34 times: systems with a global update gain little from more rollouts, and AReaL's cost of rebuilding KV caches grows with the number of trajectories and the frequency of updates.

Step 5

Staleness stays small without a bound, and the reward rises faster in wall-clock time. A rollout refreshes as soon as its batch ends or is repacked, so a trajectory's staleness reflects only how long it took to generate. Training a 7B model on 64 GPUs, it stayed typically below 3 across finish times, and the largest value in any experiment was 4. In the convergence runs (Qwen2.5-Math-7B on 32 GPUs, Qwen2.5-32B on 128, each system with its own tuned or published settings), Laminar's training reward rises 1.77 and 1.59 times faster in wall-clock time than that of synchronous verl, the best baseline; its largest staleness there was 4, the bound AReaL was given in these convergence runs (the throughput runs left AReaL unbounded). The one-step-stale and streaming pipelines gained throughput but not speed to a given reward, and AReaL, which interrupts trajectories, converged more slowly; the authors attribute this to staleness and to trajectories that mix weight versions, which distort the length distribution and break the assumption of GRPO-style methods that a group comes from one policy. Each comparison is a single training-reward curve, and whether the design changes final accuracy is not reported.

The one-step-stale and streaming baselines had higher throughput than synchronous verl but reached a given training reward later. What does this indicate?

Step 6

Separating the components confines a failure to one machine. Trajectories in progress are streamed to a partial-response pool on CPU machines, so a failed rollout's work is kept: a heartbeat detects the failure, the replica is restarted or its machine evicted, and its trajectories move to a healthy rollout with the same weight version. A failed relay is dropped from the broadcast chain, which is rebuilt in under a second, with a new master chosen if needed, and a failed trainer recovers from its checkpoint while the rollouts keep generating with the last weights. When a machine with two rollouts was killed during training, generation throughput dropped at once, training throughput stayed level or dipped slightly while the remaining rollouts filled the buffer, and both returned to their earlier levels after about 252 s, including the time to allocate and start a new machine. In systems that send weights over NCCLNVIDIA's library for collective communication among GPUs, such as broadcast and all-reduce; it has no built-in fault tolerance, one rollout failure can force the whole job to restart from a checkpoint.

Terms

Trajectory-level asynchrony: each rollout fetches new weights when its own batch ends; every trajectory comes from one weight version
Global weight synchronization: all rollouts receive new weights together after an iteration, so all wait for the slowest trajectory
Inherent staleness: $n-j$ : training steps completed while a trajectory was generated; typically below 3, at most 4 observed
Staleness and the gradient: off-policy data need the ratio $\pi_\theta/\pi_{\theta_j}$ ; clipped token ratios approximate it only while the policies are close
Partial rollout: interrupt every trajectory at an update and continue with new weights; rebuilds KV caches and mixes versions
Relay worker: CPU process per rollout machine holding the newest weights; trainer publishes once to a master relay by RDMA
Chain broadcast: $T^*(p)=WT_{\rm b}+(p-2)T_{\rm s}+2\sqrt{(p-2)WT_{\rm b}T_{\rm s}}$ , close to one copy; 72B to 127 relays in under 1.6 s
Repacking: merge the tail trajectories of rollouts with falling KV-cache use onto a few rollouts of the same version; +26% generation throughput
Strong-scaling efficiency: throughput gain divided by GPU growth; 53.7% for Laminar on math against at best 33.6%
Headline numbers: 2.56× synchronous and 1.39× AReaL throughput on average; reward 1.77× (7B) and 1.59× (32B) faster in wall-clock time

Quiz

In the chain broadcast, why is the time to reach all relays nearly independent of their number?
What do the paper's throughput numbers measure, and what do they leave out?

Sources

Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. Laminar: A Scalable Asynchronous RL Post-Training Framework. arXiv preprint, 2025.

Laminar: asynchronous RL at scale

Notation

Terms

Quiz

Sources