RolloutPipe cuts LLM training time by 30-42% with pipelined rollout
New pipelining technique keeps GPUs busy, slashing idle time by up to 76% during RL training.
Large language model post-training for reasoning tasks often uses reinforcement learning with verifiable rewards (RLVR). Modern systems adopt disaggregated architectures that separate rollout generation (where models produce candidate answers) from policy training across independent GPU pools. However, synchronous on-policy methods like GRPO make the trainer GPUs wait idle until the entire rollout completes. Asynchronous approaches overlap stages but introduce stale data, reducing training quality.
RolloutPipe addresses this by turning fixed-weight rollout into a complete-group pipeline. Two key techniques—complete-group pipelining (CGP) and frontier-group dispatch (FGD)—enable training to begin as soon as the first complete group of rollout data is ready, while later groups are still being generated. FGD prioritizes frontier groups on the rollout node to ensure steady flow of training batches. Evaluated on Qwen3-1.7B with 12 rollout configurations across four reasoning and science benchmarks, RolloutPipe cuts rollout-to-train-end time by 30.7-42.3% and reduces trainer idle time by 37-76% compared to the state-of-the-art Slime system.
- RolloutPipe introduces complete-group pipelining (CGP) to dispatch each trainable group to the trainer as soon as it materializes, eliminating idle GPU time.
- Frontier-group dispatch (FGD) prioritizes frontier groups on the rollout node so trainer-ready groups arrive earlier and more steadily.
- On Qwen3-1.7B benchmarks, RolloutPipe reduces rollout-to-train-end time by 30.7%-42.3% and lowers trainer waiting ratio by 37%-76% compared to Slime.
Why It Matters
Enables faster LLM post-training for reasoning tasks without sacrificing data freshness or policy correctness.