Researchers unveil fairness-aware LLM scheduler with 10% latency cut

As LLMs are deployed with heterogeneous workloads, chunked-prefill execution has become a mainstream serving architecture. However, existing systems rely on rigid First-Come, First-Served (FCFS) policies and static token budgets, leading to head-of-line blocking and request starvation. To address this, researchers propose a fairness-aware and latency-controllable scheduling framework. They introduce an aging-based policy that calculates dynamic priorities using accumulated waiting time and remaining prefill work. Additionally, Latency-Prediction-Based Request Scheduling (LPRS) replaces static budgets with target-time constraints, while Active Prefill Control (APC) actively regulates prefill concurrency to suppress fragmentation.

Evaluated on NVIDIA GPUs and Ascend accelerators with real-world workloads, the aging policy reduces mean end-to-end latency by over 10% compared to FCFS. LPRS and APC further significantly cut P99 tail latency and prefill fragmentation, demonstrating that structural prefill control and temporal latency constraints are complementary. The work highlights a path toward fairer, more stable LLM serving under mixed workloads. All code has been released on GitHub, providing a practical toolkit for production systems.