SMetric scheduler boosts LLM agent serving TPS by 16%

LLM serving for AI agents differs drastically from human chat: agents act only on complete responses, making cluster-wide tokens per second (TPS) the primary goal, and requests share massive KV cache reuse (over 80% in BAILIAN's production trace vs. 54–62% for chat). Current schedulers prioritize routing requests to instances that already cache their KV blocks, overloading a few machines while others sit idle, which caps overall TPS. This paper first systematically studies agent request scheduling on two real-world traces and identifies two key insights: load balance need not sacrifice all KV reuse thanks to a global-tier KV store, and leveraging intra-session locality means balancing just the first request of each agent session suffices to even out cluster load without losing most local instance reuse.

SMetric implements balanced session-centric scheduling: it routes each session's first request purely for load balance (ignoring cache locality), then routes follow-up requests in a cache-aware manner to preserve local reuse while keeping demand on the global store low. The session turn information (first vs. follow-up) serves as the scheduling metric – derived efficiently and accurately from user inputs, keeping the scheduler stateless and clean. In experiments, SMetric improves cluster TPS by 10–16% under prefill-decode colocation and prefill TPS by 2–34% under disaggregation compared to state-of-the-art schedulers, while also improving per-token latency. This work provides a practical, production-ready approach for scaling agentic LLM serving.