SmartGen cuts LLM inference latency 4.3x with selective KV cache transfer

Disaggregated LLM inference—splitting prefill and decoding across separate node pools—is a standard architecture for high-throughput serving. But for self-hosted deployments on rented cloud instances, the network link between nodes becomes a bottleneck. Shipping full KV caches (the memory buffers that store attention state) can saturate limited inter-node bandwidth, crippling time-to-first-token and overall responsiveness.

SmartGen, a new system from researchers at Fudan University, tackles this by selectively transferring only the most essential KV cache entries. It uses three coordinated paths: a profile-based proactive path that identifies and pushes critical entries during prefill, a parallel on-demand path that fetches remote and local caches simultaneously during decoding, and a speculative path that eventually delivers all remaining KV data. This layered approach makes the transfer process seamless without sacrificing output quality.

In experiments, SmartGen reduced time-to-second-token by up to 4.3x compared to conventional full KV transfer, while achieving nearly identical decoding speed and accuracy. That's a major win for teams running self-hosted LLMs on cost-effective, bandwidth-constrained cloud setups. The paper is available on arXiv (2607.28150), and the authors have released no code yet, but the architecture is ready for replication.