A recurring issue in on-policy RL for LLMs is GPU under-utilization while actors wait for weight syncs from the learner. PipelineRL uses in-flight weight updates: actors keep sampling while the learner updates weights, which reduces policy lag without stalling the pipeline.
In practice this gives ~2× wall-clock speedups on large models.
A paper on the approach was recently accepted to TMLR and discusses policy-lag bounds in more detail.
I’m involved with this project.
A recurring issue in on-policy RL for LLMs is GPU under-utilization while actors wait for weight syncs from the learner. PipelineRL uses in-flight weight updates: actors keep sampling while the learner updates weights, which reduces policy lag without stalling the pipeline.
In practice this gives ~2× wall-clock speedups on large models.
A paper on the approach was recently accepted to TMLR and discusses policy-lag bounds in more detail.
https://arxiv.org/abs/2509.19128