Director cuts MoE model latency 11-55% with proactive expert placement
New system predicts expert usage to dynamically place them across GPUs, cutting latency.
Expert parallelism is the go-to approach for serving Mixture-of-Experts (MoE) models, but its efficiency hinges on how experts are placed across GPUs. Existing placement strategies rely on historical activation patterns and struggle with diverse, rapidly changing request flows. Researchers from multiple institutions present Director, a system that tackles this with online, proactive expert placement. It predicts expert activation patterns for incoming requests using a cascaded predictor or a low-bit quantized replica, then adjusts placements accordingly.
Director's key innovations include a migration module that executes changes with near-zero downtime by aligning migrations with compute-bound phases, and a relaxation-based optimizer that runs in polynomial time while achieving a (1+ε) approximation ratio under capacity constraints. In extensive experiments, Director reduced end-to-end latency by 11–55% for popular open-source MoE models like Mistral, DeepSeek, and Qwen compared to existing work. The paper was accepted at INFOCOM 2026 and is available on arXiv.
- Director predicts expert activation patterns using a lightweight cascaded predictor or low-bit quantized replica for dynamic workload adaptation.
- Achieves near-zero downtime migration by scheduling moves during compute-bound phases, keeping disruption bounded.
- Demonstrates 11–55% end-to-end latency reduction on Mistral, DeepSeek, and Qwen MoE models compared to current methods.
Why It Matters
Efficient MoE serving cuts costs and response times for large language models, enabling faster AI at scale.