TEMPO boosts MoE serving efficiency by 15.6% with smart load balancing

A team of researchers led by Jie Li from multiple institutions has developed TEMPO, a novel dispatcher designed to optimize load balancing in expert-parallel (EP) Mixture-of-Experts (MoE) serving systems. The key innovation lies in TEMPO's ability to handle two distinct operational regimes: memory-bound (below ~156-168 tokens) and compute-bound (above that threshold). Traditional dispatchers like EPLB or METRO often fail because they assume linear relationships between tokens and expert processing time, which doesn't hold in real-world scenarios where HBM weight streaming or grouped GEMM operations dominate.

TEMPO addresses this gap by introducing a max-affine profile (t = max(a + bG, c + βN)) that accurately models both regimes. The dispatcher then solves a fixed-charge makespan problem—formalized as NP-hard in general but polynomial in degenerate cases—in milliseconds off the critical path. When integrated with SGLang as an out-of-process component, TEMPO fuses dispatch with count collection into a single in-graph kernel. Testing on an 8-GPU testbed showed TEMPO maintains within 1% of the best fixed baseline across scenarios while delivering up to 15.5% performance gains in mixed regimes. In end-to-end evaluations with Qwen3-235B, TEMPO achieved 4-6% higher throughput and reduced p99 latency by ~15.6%, though it showed minimal benefit for communication-dominated models like DeepSeek-V3.