Research reveals how to compress MoE models without losing quality
Qwen3.6-35B-A3B can lose 60% of experts and still work well, says new study
A depth-aware sensitivity analysis of the Qwen3.6-35B-A3B Mixture-of-Experts model—40 MoE layers, 256 experts per layer, top-8 routing—shows that layer sensitivity is strongly depth-dependent. Early and middle layers are highly fragile to magnitude-based expert masking, while late layers tolerate aggressive masking of low-magnitude experts. On the XLCoST cross-lingual code translation benchmark, a very-late policy masking layers 35–39 at 50% retained 419/500 Good+Similar outputs while masking only 640 of 10,240 total experts. Reducing top-k routing from 8 to 6 active experts per token also produced a large wall-clock reduction on a 100-prompt probe with no Good+Similar loss, though it does not yet compose cleanly with aggressive expert masking.
- Qwen3.6-35B-A3B (40 MoE layers, 256 experts/layer) shows depth-dependent sensitivity to expert pruning
- Late layers (35–39) tolerate aggressive masking: masking 640 experts retained 419/500 high-quality outputs
- Pruning top-k routing from 8 to 6 experts yields wall-clock speedup with no quality loss in short tests
Why It Matters
Unlocks efficient MoE compression without sacrificing performance, enabling deployment at lower cost and higher speed.