DeltaServe co-serves inference and fine-tuning, boosting throughput 2.9x

LLM serving systems are provisioned for peak traffic, leaving significant GPU compute idle during off-peak hours. DeltaServe converts that idle capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). Developed by researchers including Jiaxuan Chen and colleagues, DeltaServe integrates with existing inference engines via a compact hook interface that only requires multi-LoRA batching support. It exploits the shared execution structure between inference prefill and LoRA forward passes, plus an SLO-aware scheduler that admits fine-tuning only when sufficient inference headroom exists.

On a production trace, DeltaServe on vLLM achieved 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also delivered 39% higher fine-tuning throughput than a vLLM+torchtune baseline, using no additional hardware and maintaining full SLO compliance. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. With support for vLLM, SGLang, and S-LoRA, DeltaServe offers a practical path for GPU clusters to profitably co-locate training and serving—potentially cutting infrastructure costs for companies running large-scale LLM services.