Topology-aware KV cache transfer cuts LLM inference latency 3-18x

Disaggregated LLM inference separates prefill and decode across GPU pools, forcing the KV cache to travel between them. For a 70B model, that's 2.6 GB per request, and at production scale the aggregate demand exceeds 100 GB/s. Yet popular systems like DistServe, Splitwise, and Mooncake all treat the network as uniform, using RDMA everywhere. The problem: bandwidth between GPUs varies by up to 72x depending on physical topology — 900 GB/s over NVLink within a domain, 50 GB/s over InfiniBand across nodes, and just 12.5 GB/s over TCP across data centers. Ignoring this variation leaves massive performance on the table.

Ganjihal's paper designs a topology-aware transfer orchestrator that discovers the interconnect hierarchy at startup and picks the optimal transport for each transfer. Three mechanisms work together: pipelined layer-by-layer transfer overlaps KV cache transmission with ongoing prefill computation, hiding 60 to 85 percent of latency; NVLink domain-aware placement for Mixture-of-Experts models co-optimizes expert dispatch with KV cache locality; and CXL 3.0 memory expanders act as a shared overflow tier, offering 6x capacity at 86x lower latency than NVMe. Full evaluation requires multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware, which isn't yet available in GPU clouds. But analytical models and projected analysis across three architectures show 3 to 18x transfer latency reduction over uniform RDMA.