KServe study finds 208x faster cold starts using OCI model delivery in Kubernetes

In Kubernetes inference serving, cold-start latency is dominated by delivering model weights, especially for large language models (LLMs) reaching hundreds of gigabytes. Current practice relies on ad-hoc downloads from object storage (S3, GCS, Hugging Face) that bypass Kubernetes’ built-in pull caching, digest addressing, and verification. This arXiv study by Georgii Kliukovkin benchmarks three delivery paths in KServe: modelcar sidecars, native OCI image volumes (KEP-4639, merged upstream), and OCI fetch (under review). On artifacts sized to 1B, 7B, and 70B parameters (2–140 GB), the results are stark.

The key finding: node-cached OCI delivery makes warm replica addition size-independent. A 70B-class model loads in 11.7 seconds via OCI versus 40.7 minutes via object storage—a 208x speedup. However, the first cold pull can cost up to 2x a plain download due to containerd’s blob-write-then-unpack double pass. For models on s3://, gs://, or hf:// URIs where no admission-time verifier observes bytes, the paper presents a serving-time integrity design: digest pinning and OpenSSF model-signing enforcement in the storage initializer. Streaming hash verification during download adds under 0.1% to delivery time, while a post-download pass adds up to 53%. This work, submitted to IEEE Access, has implications for autoscaling economics and trust in AI deployments.