New MoE framework cuts parameters 80% via dynamic expert clustering

Mixture-of-Experts LLMs have long suffered a trilemma of load imbalance, parameter redundancy, and communication overhead. Researchers now present a unified framework that dynamically clusters experts during training using an online clustering procedure based on a fused metric of parameter and activation similarity. This stabilizes expert utilization and enables structural efficiency gains. Within each cluster, expert weights are decomposed into a shared base matrix stored in FP16 and extremely low-rank residual adapters stored in INT4. This achieves up to fivefold parameter reduction per group while preserving specialization. A two-stage hierarchical routing strategy first assigns tokens to a cluster, then to specific experts, drastically shrinking the routing search space and reducing all-to-all communication volume.

Evaluated on GLUE and WikiText-103, the framework matches the quality of standard MoE models while reducing total parameters by approximately 80%, improving throughput by 10% to 20%, and lowering expert load variance by a factor of over three. The heterogeneous precision scheme and dynamic offloading of inactive clusters also cut peak memory to levels comparable to dense models. The paper, under review for ICML 2026, provides code and demonstrates that structural reorganization is a principled path toward scalable, efficient MoE LLMs. Key innovations include leveraging the router’s semantic embeddings for dynamic architecture reconfiguration and combining structured compression with hierarchical routing to bypass traditional trade-offs.