New MoE scheduling technique speeds up LLM inference by 2.64x

Efficient deployment of Mixture-of-Experts (MoE) models, critical for trillion-parameter LLMs, requires distributed execution where every MoE layer performs two all-to-all communications: dispatching tokens to expert ranks and returning outputs. Traditional implementations wait for expert compute to finish before starting the return all-to-all, leaving GPUs idle and dragging down utilization. The new approach from Linnaeus University solves this by overlapping expert compute with the second all-to-all via tile-level signaling and scheduling.

The co-design features a persistent per-rank computation kernel that covers all local experts, eliminating repeated kernel launch overhead and prioritizing remote-critical tiles. A persistent communication kernel runs on a small dedicated partition of streaming multiprocessors (SMs), issuing segment-granular transfers as tiles become ready. Tested on a 4-A100 GPU system across three MoE models compared to four state-of-the-art systems, the technique achieves up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup without intrusive changes to underlying operators or communication primitives. It consistently improves performance across varying GEMM shapes, router modes, and producer/consumer SM partitions while preserving correctness.