ForeMoE speeds up MoE RL post-training by 45% with micro-step load balancing

Mixture-of-Experts (MoE) architectures and reinforcement learning (RL) post-training are now standard for large language models (LLMs), but expert load imbalance remains a major performance bottleneck. Traditional load-balancing systems are designed for pre-training and rely on historical step-level statistics. However, RL post-training introduces unique workload dynamics: while step-level load appears stable, the tiny batch sizes processed during micro-steps cause severe, high-frequency fluctuations that existing methods cannot handle. This leads to significant inefficiencies and longer training times.

To solve this, researchers from Peking University and Alibaba Group developed ForeMoE, a micro-step-level load balancing system. ForeMoE leverages the multi-stage RL pipeline (rollout, recompute, policy update) by using foreseeable routing information from the rollout stage to proactively guide load balancing in the subsequent stages. For frequent per-micro-step reconfiguration, it employs a hierarchical planner that decomposes the NP-hard load balancing problem into tractable sub-components, and a transfer engine that uses complementary hardware paths (CPU-assisted and GPU-direct) for overlapped expert transfers. Evaluations on 64 GPUs show up to a 1.45× speedup over state-of-the-art RL post-training systems, making it a practical solution for scaling MoE models with RL fine-tuning.