BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization

Researchers have unveiled BOute, a new scheduling system that uses multi-objective Bayesian optimization to dramatically reduce the cost of serving large language models. It intelligently routes simple queries to cheaper, smaller models and complex ones to more powerful models, while also optimizing deployment across different GPU types. The system outperforms current state-of-the-art serving systems by 59% on average, reducing serving costs by 15-61% (38% average) while maintaining performance targets.