CrossPool cuts LLM serving latency 10x via memory disaggregation

Emerging LLM services host many sparse MoE models, but most receive few requests and stay 'cold.' This creates a GPU memory dilemma: model weights are stable and predetermined, while KV-cache is transient and demand-driven. Since cold models rarely hit peak KV-cache demand simultaneously, reserving worst-case capacity per model wastes memory. A shared KV-cache pool helps, but when weights and KV-cache remain in a monolithic GPU memory pool, static weights compete with dynamic KV-cache. Moreover, cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support.

CrossPool tackles this by disaggregating FFN weights and KV-cache into two separate GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local. It introduces a KV-cache planner and virtualizer for efficient allocation, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU overhead. The result is a serving engine that powers bursty long-context requests, outperforming the kvcached-based multi-LLM system with up to a 10.4x reduction in P99 TBT.