SlidingServe boosts LLM throughput 30% with smarter scheduling

Large language model online services face a tough trade-off: maximizing throughput while meeting strict latency SLOs. Existing systems rely on coarse-grained output constraints, leading to inefficient resource contention and limited QoS differentiation. SlidingServe, from researchers at Sun Yat-sen University, tackles this with a sliding-window-driven, SLO-aware approach. Its lightweight batch latency predictor estimates batch execution time to guide decisions. SlidingChunker dynamically adjusts chunk sizes by combining information from current and next iterations, improving throughput while maintaining QoS. A Multi-Level Priority Sorter balances fairness and efficiency, and BatchConstructor uses dynamic programming to select requests for execution when multiple are at risk of SLO violation.

In evaluation, SlidingServe outperformed state-of-the-art schedulers across load conditions. Service capacity increased by up to 30%, and under heavy load, SLO violation rates dropped 16–53%. The system's fine-grained control over batches and priorities allows it to adapt to real-time demands without manual tuning. This research, published on arXiv (2606.05933), offers a practical path for deploying high-performance LLM inference at scale. For engineers building production AI systems, SlidingServe’s techniques — especially the predictive batch latency model and dynamic programming scheduler — are immediately applicable to reduce infrastructure costs and improve user experience.