StickyMoE cuts expert switching by 60% for efficient edge AI inference
New training method keeps MoE experts 'sticky' across tokens, slashing memory swaps.
Mixture-of-Experts (MoE) models are powerful but suffer on edge devices because consecutive tokens often activate different experts, forcing constant weight swapping between slow storage and fast memory. Existing fixes—caching heuristics or post-hoc router fine-tuning—only address symptoms, not the root cause. StickyMoE, proposed by Ali Kayyam, tackles this at training time with a differentiable routing consistency loss that encourages the router to keep the same expert assignment across semantically coherent token spans.
StickyMoE requires zero architectural changes and adds just one hyperparameter lambda. In small-scale MoE language model experiments, it slashes expert switch rates by up to 60% while degrading perplexity by less than 4%, outperforming post-hoc fine-tuning on the quality-locality frontier. This training-time approach lets expert representations and routing decisions co-adapt from the first step, making it a practical, drop-in solution for deploying large MoE models on memory-constrained hardware.
- Differentiable routing consistency loss penalizes abrupt expert switches between adjacent tokens
- Reduces expert switch rate by up to 60% with less than 4% perplexity degradation
- No architectural changes—just a single hyperparameter lambda; works at training time
Why It Matters
Makes large MoE models practical on edge devices without costly hardware upgrades or post-hoc tuning.