Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

A team of researchers has introduced 'Expert Upcycling,' a novel method designed to make scaling Mixture-of-Experts (MoE) language models significantly more efficient. MoE architectures, used in frontier models like GPT-4 and Claude 3 Opus, separate total parameters from active computation via sparse routing. While scaling laws show quality improves with more parameters, training these massive, sparse models is prohibitively expensive due to memory and communication costs scaling with total parameter count. Expert Upcycling tackles this by starting with a trained model with E experts and 'upcycling' it to an mE-expert model through a clever duplication process.

The core technique involves duplicating existing experts and extending the router while keeping the top-K routing fixed, which preserves the per-token inference cost. This duplication provides a warm start; the expanded model inherits the source model's knowledge, beginning training at a much lower loss than random initialization. Subsequent continued pre-training (CPT) then breaks the symmetry among the duplicated experts, allowing them to specialize in different tasks or knowledge domains. The researchers also developed 'utility-based expert selection,' which uses gradient importance scores to guide which experts to duplicate, more than tripling the efficiency of gap closure when training time is limited.

In practical experiments across 7B to 13B total parameter scales, the upcycled models achieved the same validation loss as models trained from scratch at the larger size, but did so using 32% fewer GPU hours. The paper provides comprehensive ablations and a practical recipe for deploying the method, establishing Expert Upcycling as a principled and compute-efficient alternative for organizations looking to build larger, more capable MoE models without the astronomical training budgets typically required.