New ElastiCo framework cuts GPU cluster costs 3x by sharing resources
ElastiCo's interference-aware GPU sharing boosts cluster throughput 2x while cutting costs by 50%.
A team of researchers from Peking University and the University of Sydney has unveiled ElastiCo, a breakthrough framework designed to optimize GPU cluster utilization by intelligently co-locating deep learning training and inference workloads. Published on arXiv, the work addresses a critical inefficiency in modern GPU clusters: traditional schedulers isolate resources for training and inference jobs, leading to chronic underutilization as training jobs reserve entire GPUs despite periodic idle phases, while inference tasks over-provision resources for bursty demand.
ElastiCo introduces three core innovations to solve this problem. First, 'Resource Shape Transformation' allows each job to operate across multiple feasible resource-performance configurations. Second, 'Elastic Shadow Pricing' dynamically adjusts per-resource prices to decompose complex allocation problems into simpler, per-job configuration choices. Finally, the framework's 'Interference-Aware Co-location' model predicts performance degradation from GPU sharing using hardware counters and task-level features, enabling safe multi-tenant operation. Implemented as native Kubernetes middleware without requiring code changes, ElastiCo demonstrated dramatic improvements in real-world tests: a 64-GPU testbed and simulations up to 512 GPUs showed a 2.94x reduction in average job completion time (JCT), a 2.02x increase in cluster throughput, and a near-doubling of GPU utilization from ~25% to 46%.
- ElastiCo is a Kubernetes-native framework from Peking University and University of Sydney that co-locates training and inference workloads on shared GPUs without code changes
- Demonstrated 2.94x reduction in job completion time and 2.02x throughput increase in 64-512 GPU cluster tests, with GPU utilization jumping from 25% to 46%
- Uses dynamic resource pricing and interference prediction models to safely maximize cluster efficiency
Why It Matters
Cuts GPU cluster costs by 50% while accelerating AI workloads—critical for scaling LLM inference and training efficiently.