Gimbal's coordinated scheduling slashes MoE LLM latency by 42%

Serving Mixture-of-Experts LLMs is notoriously difficult because dynamic request patterns create imbalances across data-parallel engines and expert-level hotspots. Existing systems like vLLM treat frontend scheduling and backend expert balancing as separate problems, missing the fine-grained pressure signals needed for optimal decisions. Gimbal bridges this gap with a dual-layer approach: a frontend scheduler that considers live backend metrics—KV cache usage, remaining prefill work, queue depth, and per-expert load—to route requests intelligently, and a per-engine queue ordering policy that reduces head-of-line blocking without needing output-length prediction. On the backend, Gimbal extends expert load balancing by tracking source-DP-to-expert routing patterns and uses a mixed-integer nonlinear program (MINLP) heuristic to place experts while balancing load, communication costs, and migration stability.

In benchmark evaluations, Gimbal outperforms vLLM by a significant margin: average Time To First Token (TTFT) drops 42.9%, average Time Per Output Token (TPOT) falls 33.3%, and high-load request throughput increases by 3.0%. These gains come without compromising model quality or requiring changes to the underlying MoE architecture. For organizations running LLMs at scale—especially those using MoE models like Mixtral or deepseek—Gimbal offers a practical path to lower latency and higher throughput. The system is designed to be integrated into existing serving stacks, making it viable for production deployments.