BiDiRL boosts RL post-training throughput by 1.94x with bidirectional scheduling
New hybrid architecture eliminates idle GPU resources during asynchronous RL training for LLMs.
Large language model reasoning capabilities improve with reinforcement learning (RL) post-training, but standard asynchronous, disaggregated architectures (separating rollout and training resources) often leave GPUs idle due to workload imbalances. This paper from Tan Zhiqiang, Wang Maoxin, and colleagues introduces BiDiRL, a novel hybrid time-space multiplexing framework designed to minimize that idle time.
BiDiRL combines three key innovations: a hot-switch runtime that rapidly repurposes rollout resources for training (or vice versa) with negligible overhead, a scheduling-aware planner that pre-partitions resources to balance durations at a coarse level, and a real-time bidirectional scheduler that dynamically borrows idle resources from the non-bottleneck pool. In extensive tests on two 32-GPU clusters across various models, datasets, and hyperparameters, BiDiRL delivered up to a 1.94x throughput improvement over existing RL systems like veRL, AReaL, and ROLL—while maintaining identical convergence behavior. This means faster, cheaper RL-based fine-tuning for advanced LLMs without any quality trade-off.
- Hot-switch runtime enables rapid switching between rollout and training resources with near-zero overhead (under 1 ms latency).
- Static scheduling planner uses time-performance modeling to pick a resource partition that balances rollout and training durations.
- Bidirectional scheduler exploits runtime bubbles, allowing the bottleneck stage to temporarily borrow idle GPUs from the other pool, boosting throughput by up to 1.94x on 32-GPU clusters.
Why It Matters
Slashing idle GPU resources in RL post-training cuts costs and speeds up LLM reasoning improvements—critical for scaling advanced AI.