Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

As LLM usage fragments across countless models, cloud providers offer a wide range of mid-tier and older-generation GPUs that often deliver comparable performance per dollar to top-tier hardware but remain underutilized. Enter Coral, a new serving system designed by researchers from Carnegie Mellon University and UC Berkeley to efficiently harness these heterogeneous resources for concurrent multi-LLM deployments. Coral’s key innovation is a joint optimization of resource allocation and serving strategy across all model replicas, treating the entire cluster as a single scheduling problem rather than optimizing each model independently. This holistic approach allows Coral to adapt to shifting throughput demands and resource availability in real time.

To make this joint optimization practical, Coral applies a lossless two-stage decomposition that preserves optimality while slashing solve time from hours to just tens of seconds. In evaluations across six popular LLMs and 20 distinct GPU configurations, Coral reduced serving costs by up to 2.79× over the best baseline systems (including state-of-the-art schedulers like Shepherd and MUX). Under scarce resource conditions, Coral delivered up to 2.39× higher goodput—meaning more successful requests completed on time. The system is particularly valuable for organizations running multiple LLMs (e.g., GPT-3.5, Llama 3, Mistral) on spot or preemptible cloud instances, where resource availability fluctuates. Coral's source code and benchmarks are expected to be released, enabling cloud operators to dramatically lower inference costs without sacrificing performance.