InferScale's GPU-native KV injection slashes LLM serving latency by 4.8x

Large language models (LLMs) deployed with persistent personalized context—like memory profiles or long conversation histories—face a scalability bottleneck: every request forces the serving engine to repeatedly prefill the same retrieved memory content. This inflates time-to-first-token (TTFT) as the retrieval budget grows, even though the underlying memory data is reused across a user's many requests.

InferScale, developed by Peter Li and Prashant Pandey, solves this by precomputing each memory fact's key-value (KV) representation, storing it alongside a semantic embedding directly on the GPU. At serving time, it retrieves relevant facts and injects their KV state into vLLM's paged cache. To handle dynamically assembled memories under rotary position embeddings (RoPE), it introduces Chunked RoPE—storing keys before rotation and applying serving-time positions during injection. A companion technique, Context-Window Encoding, encodes each memory fact with a small window of preceding conversation to recover cross-fact context lost by independent encoding. Across three open-weight models on the LoCoMo benchmark, InferScale keeps TTFT nearly constant as retrieval budget increases: at k=50 it reduces TTFT by 72–79% (3.6–4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 (without serving-time recomputation), and yields 3.7–4.5x the throughput under concurrent load. Crucially, it requires no engine modifications or model fine-tuning—just vLLM's KV-connector interface.