New scheduler cuts LLM tail latency 81% by balancing prefill and decode
Researchers propose load-aware prefill deflection to eliminate KV-cache transfer bottlenecks.
Disaggregated LLM serving separates prefill and decode onto different GPU pools to avoid interference, but under bursty workloads prefill nodes often saturate while decode nodes sit idle. This imbalance causes queuing delays and expensive inter-node KV-cache transfers, which can account for 77–98% of P95 time-to-first-token (TTFT) in production-style 2P2D A100 clusters.
To solve this, the authors present a proactive, load-aware scheduler that allows decode nodes to opportunistically serve prefill chunks alongside their in-flight decode batches. The scheduler estimates the TTFT for each queued request on the prefill node and on every decode node, then picks the largest chunk schedule that keeps decode latencies within SLO. By running the prefill phase locally on the decode node, inter-node KV-cache transfer is eliminated entirely, and the approach is implemented on vLLM using DeepSeek-V2-Lite traces. It achieves up to 81% lower P95 TTFT and 79% higher SLO attainment compared to state-of-the-art disaggregated schedulers, with negligible routing cost.
- Queuing and KV-cache transfer cause 77–98% of P95 TTFT under bursty workloads, not prefill execution itself.
- Proactive scheduler defers prefill to underutilized decode nodes, interleaving chunked prefill with decode batches.
- Implemented on vLLM with DeepSeek-V2-Lite, achieving up to 81% lower P95 TTFT and 79% higher SLO attainment.
Why It Matters
For LLM serving infrastructure, this could slash latency without adding GPUs, improving user experience and cost efficiency.