Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock

AWS has announced a significant optimization for hosting multiple fine-tuned AI models, developed in collaboration with the open-source vLLM community. The new Multi-LoRA (Low-Rank Adaptation) serving capability specifically targets the challenge of costly, underutilized GPU capacity when organizations deploy numerous custom models, particularly from the emerging Mixture of Experts (MoE) families like GPT-OSS and Qwen. Instead of dedicating a GPU to a single, sporadically-used model, this solution allows dozens of models to share the same hardware by keeping the base model weights frozen and dynamically loading only the small, trained LoRA adapters for each inference request. This turns what would be five underutilized GPUs into one efficiently shared resource, dramatically improving cost efficiency for custom AI deployments on Amazon SageMaker AI and Amazon Bedrock.

The technical breakthrough involves kernel-level optimizations within vLLM to efficiently handle the fused operations required for MoE architectures. For a model like GPT-OSS 20B, AWS-specific enhancements deliver a 19% increase in Output Tokens Per Second (OTPS) and an 8% reduction in Time To First Token (TTFT) compared to the standard vLLM 0.15.0 release. While built for sparse MoE models, the improvements also benefit dense models like Llama3.3 70B. This move directly addresses a major pain point in enterprise AI: the prohibitive cost of serving many specialized, low-traffic models. It empowers companies to deploy a fleet of fine-tuned models for different tasks or departments without multiplying their infrastructure spend, making customized AI more accessible and scalable.