Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
New method fixes performance degradation in compressed MoE models by updating only 0.1% of parameters.
A new research paper from Sieun Hyeon and Jaeyoung Do tackles a major deployment bottleneck for massive Mixture-of-Experts (MoE) models like Mixtral 8x7B. While MoE architecture efficiently scales model capacity, its huge parameter count creates memory constraints. The study systematically analyzes three retraining-free compression methods—Expert Pruning, Editing, and Merging—and identifies a previously overlooked culprit for performance degradation: router-expert mismatch. When experts are compressed or altered, the original router becomes misaligned, leading to poor token routing decisions. The authors argue that truly effective compression must avoid costly full-model retraining while allowing for lightweight router adjustment.
To solve this, the team proposes Router Knowledge Distillation (Router KD), a method that updates only the router's parameters—a tiny fraction (often <0.1%) of the total model—by distilling the original model's next-token predictions on a small set of unlabeled calibration data. Experiments show Router KD consistently recovers performance across all compression paradigms, with particularly large gains for fine-grained MoEs (with many small experts) due to their more complex routing boundaries. This work provides a practical, low-cost pathway to deploy compressed, high-performance MoE models in resource-constrained environments, potentially making advanced models more accessible.
- Identifies router-expert mismatch as the key cause of performance loss in compressed MoE models, a flaw overlooked in prior retraining-free methods.
- Proposes Router Knowledge Distillation (Router KD), which updates less than 0.1% of model parameters (the router) using unlabeled data, avoiding full retraining.
- Shows consistent performance recovery across Expert Pruning, Editing, and Merging, with up to 2x larger gains in fine-grained MoEs versus coarse-grained ones.
Why It Matters
Enables efficient deployment of massive MoE models (like Mixtral) on consumer hardware by fixing compression flaws without expensive retraining.