OpScale cuts LLM serving GPU needs by 36% with operator-level autoscaling

A new arXiv paper from researchers including Xingqi Cui and Chieh-Jan Mike Liang introduces OpScale, a framework for provisioning and autoscaling LLM serving workloads at the operator level. Traditional serving systems treat the entire LLM as a monolithic scaling unit, which is simple but fails to capture fine-grained inference dynamics—leading to SLO violations under bursty demand or severe GPU under-utilization. OpScale instead treats individual operators within the model as the scaling primitive, enabling finer-grained resource allocation that matches the heterogeneous compute needs of attention, feed-forward, and other layers.

OpScale addresses the inherent complexity and space explosion of operator-level scheduling with a four-stage pipeline: profiling, provisioning, placement, and runtime serving. It automatically profiles each operator's resource footprint, provisions GPU capacity per operator, places them optimally across devices, and adjusts allocations in real time. The researchers evaluated OpScale on production workloads using up to 40 A100s and 24 GB200s. Results show it achieves SLOs with up to 36.3% fewer GPUs and 28% less power compared to monolithic scaling—or 44% higher throughput under a fixed cost budget. This could significantly lower the operational overhead for cloud providers and enterprises running large-scale LLM inference, especially in dynamic, unpredictable traffic environments.