MCF-MOE improves MoE expert selection with multi-level context fusion

A new paper from researchers including Shuhan Huang and colleagues tackles a core weakness in Mixture-of-Experts (MoE) architectures: inconsistent expert selection. Standard MoE routers rely on shallow, isolated token representations, causing unstable routing that undermines specialization. The team identifies "context incompleteness" as the primary bottleneck and proposes Multi-level Context Fusion MOE (MCF-MOE). Their framework constructs richer representations by combining cross-layer semantic aggregation—pulling signals from multiple transformer layers—with local token-level interactions. This dual-fusion approach ensures each expert sees a more complete picture of the input, enabling routing decisions that stay consistent across layers and examples.

Evaluated on language modeling and understanding benchmarks, MCF-MOE consistently beats strong MoE baselines, including standard top-k routing and earlier context-aware methods. The improvements in routing stability translate directly to downstream performance gains without significant computational overhead. By showing that context depth and breadth matter for expert routing, this work provides a practical upgrade for scaling large language models. The researchers have open-sourced the code, making it easy for the community to integrate MCF-MOE into existing transformer training pipelines.