ELDR: New routing cuts MoE LLM serving latency by up to 13.9%
KAIST and Microsoft researchers slash decode time with expert-locality-aware routing in vLLM.
In PD-disaggregated LLM serving, prefill and decode stages run on separate workers, and each request must be routed to a decode worker after prefill. Traditional decode routers only balance load by token count or queue length, but for Mixture-of-Experts (MoE) models this is incomplete. Equally loaded workers can have vastly different latencies because each decode step must load the weights of every distinct expert activated by the batch. The paper introduces ELDR (Expert-Locality-Aware Decode Routing), which exploits the observation that expert activations are highly correlated between prefill and subsequent decode steps.
ELDR builds an 'expert signature' for each request during prefill, then uses offline balanced K-means clustering to partition the signature space across decode workers. Online, a locality-band routing scheme sends each request to the least-loaded worker among those whose cluster best matches its signature. A signature cache, co-indexed with the KV cache at the granularity of KV blocks, keeps signatures exact even under prefix caching. The system is implemented in vLLM and evaluated on up to 40 GPUs with three MoE models (including Mixtral-8x7B and Qwen2-57B-A14). Results show median TPOT reduction of 5.9–13.9% over the strongest of four load-balancing baselines, with no changes to model outputs.
- ELDR reduces median time-per-output-token by up to 13.9% vs. the best load-balancing baseline across three MoE models on 40 GPUs.
- Uses expert signatures from prefill activations to predict decode-time expert usage, then routes via balanced K-means clustering and locality-band assignment.
- Implemented in vLLM with a signature cache co-indexed to KV blocks, enabling exact routing even under prefix caching.
Why It Matters
ELDR tackles a hidden inefficiency in MoE serving, offering a drop-in latency improvement for production LLM deployments.