Amazon SageMaker AI container caching slashes scale-out latency by 51%

Amazon SageMaker AI has announced container image caching for inference endpoints, targeting a major bottleneck in scaling generative AI workloads. When new instances are launched during scale-out events, the container image pull from Amazon ECR often dominates startup latency—especially for large containers like LMI (powered by vLLM), vLLM, and NVIDIA Triton, which can exceed 17 GB compressed. Previously, SageMaker’s inference component data caching helped only when reusing existing instances. Now, container caching stores images on newly provisioned instances, completely removing the pull step. In a benchmark using the Qwen3-8B model (16 GB) on an ml.g6.2xlarge with an LMI container (17.7 GB compressed), startup latency fell from 525 seconds to 258 seconds—a 51% improvement. The cache also eliminates network bandwidth contention between image download and model artifact fetch, further speeding up model download from 168s to 77s. If a cached image is unavailable, SageMaker automatically falls back to pulling from ECR, ensuring scaling is never blocked.

Container caching works seamlessly with both single-model endpoints and inference component-based deployments, where multiple models share instances. Each unique container image referenced by inference components is cached, and strict tenant isolation is maintained—cache is dedicated to a single account and endpoint. This feature builds on earlier SageMaker innovations like sub-minute CloudWatch metrics (6x faster detection) and inference component data caching. Together, these capabilities deliver comprehensive faster scaling optimization for generative AI, reducing cold starts during demand surges. For practitioners deploying large models, this translates directly to lower latency variability and cost, as fewer instances need to sit idle waiting for scale-out to complete.