ReMP framework reconfigures LLM parallelism in 1-7 seconds without restart
New technique slashes model reconfiguration downtime from minutes to seconds.
Current LLM inference systems rely on static TP and PP configurations, but real-world workloads fluctuate—causing a fundamental mismatch. Restarting to change topology causes minutes of downtime, KV cache loss, and costly recomputation. ReMP, presented in a new arXiv paper by Haipeng Yuan and colleagues, solves this with a runtime reconfiguration framework that keeps services online. Three key innovations underpin ReMP: first, it decouples the parallelism topology from runtime state to avoid full reconstruction; second, it introduces a two-dimensional KV cache migration mechanism that preserves reusable cache during TP/PP changes; third, it implements end-to-end online reconfiguration without service interruption.
Experiments show ReMP completes most topology shifts in just 1-7 seconds across 7B-70B models, achieving tens to over a hundred times speedup versus restarting. Under dynamic loads, ReMP consistently outperforms fixed configurations on latency metrics like TTFT and TPOT, while boosting output throughput. This breakthrough allows LLM serving systems to adapt on the fly to changing request patterns, reducing wasted compute and improving user experience—a significant step toward elastic, efficient AI infrastructure.
- ReMP completes model-parallelism reconfiguration in 1-7 seconds for 7B-70B parameter models, vs. minutes for restart-based approaches.
- Achieves tens to over 100x speedup compared to service restart, thanks to decoupled topology and runtime state.
- Preserves KV cache via 2D migration, eliminating recomputation overhead and improving TTFT, TPOT, and throughput under dynamic workloads.
Why It Matters
Enables elastic LLM serving that adapts to traffic spikes without downtime or wasted compute.