InstantInfer speeds up LLM cold starts 7.2x with new CFA framework
A new technique cuts LLM cold start time dramatically by refactoring sequential processes.
Cold starts in large language model (LLM) inference services are notoriously slow due to sequential initialization and massive fine-grained I/O requests from complex software stacks. Researchers from Peking University and Scitix AI introduce InstantInfer, built on a Communicating Finite Automata (CFA) abstraction that systematically identifies cross-component optimization opportunities. Their programming framework preserves the original sequential program structure while enabling safe concurrent execution, eliminating the correctness risks of manual refactoring.
InstantInfer is demonstrated by refactoring three key components of vLLM: process tree creation, tensor loading, and model switching. Experiments show up to 7.2x speedup during cold starts, and the system remains robust across different GPU architectures, workload patterns, and cluster scales. This work provides a principled approach to accelerating LLM inference initialization without sacrificing reliability—a critical need for production deployments where low latency and high throughput are paramount.
- Achieves up to 7.2x speedup in LLM cold starts using a Communicating Finite Automata abstraction.
- Enables safe refactoring of sequential vLLM components into concurrent execution.
- Demonstrates robustness across diverse GPUs, workloads, and scales in extensive experiments.
Why It Matters
7x faster LLM cold starts could drastically reduce latency and improve user experience in production inference services.