RoutePack boosts MoE RL training throughput by 14.9%
New system cuts Mixture-of-Experts RL training time by up to 14.9% with expert rerouting and data packing
Deep Dive
RoutePack is a new hierarchical planner for Mixture-of-Experts (MoE) reinforcement learning that coordinates expert rerouting with attention-aware data packing. On the Ling-3.0-Tiny and Ling-3.0-Flash models, it improves trainer-measured token throughput by 8.85% and 14.89% over baseline.
Key Points
- RoutePack improves MoE RL training throughput by 14.9% on Ling-3.0-Flash through expert rerouting and data packing
- Achieves 3.8-10.5% gains from expert placement alone and 4-5% from data packing optimization
- Uses state-consistent materialization to preserve existing MoE kernels without replication
Why It Matters
Cuts distributed RL training costs for large MoE models, enabling faster iteration on cutting-edge AI systems