New MoE pruning method cuts memory 5.27x without accuracy loss
Researchers find MoE layers have hidden channel-level redundancy, enabling extreme compression.
Mixture-of-Experts (MoE) models scale compute efficiently but remain memory-heavy. Existing compression methods remove entire experts or rank them by coarse importance, missing fine-grained redundancy inside individual experts. A new paper from researchers at multiple institutions proposes Attribution-Guided and Coverage-Maximized Pruning (AGCP), a structural pruning framework that operates at the channel level within experts. The key insight: information within MoE experts is concentrated in a small subset of channels. AGCP reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation.
Experiments on DeepSeek and Qwen MoE models show AGCP preserves accuracy under 50% or even 25% structured pruning when combined with 4-bit quantization. On the Qwen3-30B-A3B model, the method reduces memory footprint by 5.27× and consistently outperforms state-of-the-art baselines across multiple benchmarks. This channel-level approach enables much more aggressive compression without sacrificing performance, potentially making large-scale MoE models practical for deployment on resource-constrained devices. The paper is submitted to ICML 2026.
- Method prunes channels within MoE experts, not entire experts, capturing fine-grained redundancy.
- Achieves 50% structured pruning with 4-bit quantization without accuracy loss on DeepSeek and Qwen models.
- Reduces memory footprint of Qwen3-30B-A3B by 5.27x, outperforming state-of-the-art baselines on diverse benchmarks.
Why It Matters
Enables much smaller and faster MoE models for deployment, potentially making large-scale MoE practical for edge devices.