GeMoE: New routing method boosts MoE efficiency by 36% sparsity

Mixture of Experts (MoE) architectures let large vision-language models (LVLMs) scale parameters without proportional inference cost increases. But the standard Top-k static routing assigns the same number of experts to every token—wasting compute on simple tokens and underpowering complex ones. Researchers from Zhejiang University now propose GeMoE (Gating Entropy-based Uncertainty-aware Adaptive Routing), which treats token routing as an information encoding task. By framing dynamic expert selection as a Minimum Description Length (MDL) problem, they link gating entropy directly to token complexity. The result: each token engages exactly as many experts as needed, no more, no less.

Across a range of backbones and benchmarks, GeMoE achieves 99.5% of the static routing's performance while boosting expert activation sparsity by 36.5%. That means roughly a third fewer expert calls per token on average—translating to faster inference and lower memory bandwidth. The method is model-agnostic and requires no retraining of the base MoE. For practitioners running LVLMs at scale, GeMoE offers a principled, plug-and-play way to cut costs without sacrificing accuracy.