AgentSysBench exposes six ways agentic workloads break LLM serving

As AI applications shift from single LLM inference calls to long-running, agentic workloads that coordinate tools, environments, and persistent state, the assumptions behind conventional serving systems are breaking down. To quantify this, Chaokun Chang and 12 co-authors introduce AgentSysBench, a benchmark suite with ten representative agentic applications and unified system-level instrumentation. Their analysis of controlled deployments and production traces identifies six distinguishing properties: execution is heavyweight and stateful, with non-LLM components dominating latency in 5 of 10 applications and sandbox working-set memory peaking at 28GB per session; workloads compose heterogeneous resources—GPU-bound inference, memory-bound retrieval, CPU-bound sandboxes—whose latencies diverge by up to 32x; bottlenecks shift unpredictably; sessions hold state idle for minutes to hours; auxiliary LLM calls and tool-schema context create a control-plane tax; and production traces reveal heavy cross-request redundancy in searches and web fetches.

These findings translate directly into actionable optimizations. Task-aware serving reduces latency by 29-40%, and communication-aware placement speeds up handling by up to 4.5x. State offloading cuts memory consumption 4.6x, while tool-result caching eliminates 35.2% of redundant search calls and saves 19.3% of aggregate search latency. The paper, available on arXiv (2608.15127), argues that serving systems need architectural changes—not just better GPU scheduling—to handle agentic workloads efficiently. For platform engineers, this means treating CPU sandboxes, memory-bound retrieval, and auxiliary LLM calls as first-class scheduling resources, and building caching into the control plane. The 4.6x memory reduction alone could be the difference between renting one node or five for production agent deployments.