ShuntServe cuts LLM serving costs 31% using spot GPUs
Mix cheap spot GPUs for 90% savings without losing performance.
As LLM serving costs skyrocket, spot GPU instances offer up to 90% savings but suffer from frequent interruptions and limited availability. ShuntServe tackles this by allowing heterogeneous spot clusters—mixing GPU types like L4, A10G, and L40S—to leverage complementary availability patterns. It uses a roofline model-based performance estimator and a dynamic programming optimizer to jointly decide node configuration, parallelization strategy, and layer assignment, maximizing throughput across mismatched hardware.
Fault tolerance is handled via output-preserving request migration and concurrent initialization through a shared tensor store, minimizing downtime during spot instance preemption. Tested on AWS with Llama-3.1-70B and Qwen3-32B, ShuntServe delivered 1.42x and 1.35x throughput gains over state-of-the-art baselines, with 31.9% and 31.2% cost efficiency improvements for offline and online serving, respectively. The paper (arXiv:2606.18600) includes 18 pages of detailed analysis and benchmarks.
- Uses heterogeneous spot GPU clusters (L4, A10G, L40S) to reduce costs by up to 90% vs. on-demand.
- Roofline model + dynamic programming optimizer increases throughput 1.42x over baselines.
- Achieved 31.9% cost efficiency improvement for offline serving and 31.2% for online serving.
Why It Matters
Makes enterprise LLM serving dramatically cheaper by reliably using cheap spot instances across diverse GPU types.