DualDecoder speeds long-context LLM inference by up to 2.62x

Long-context LLM inference faces a severe memory wall as the KV cache scales with context length and concurrency. Existing sparse KV cache methods offload entries to host memory but require auxiliary GPU states for retrieval management, creating a new bottleneck under high concurrency. DualDecoder, presented by Zuning Liang et al., offers a solution by predicting the critical KV entries needed for the next token from the preceding speculated token. This enables proactive prefetching from host memory that overlaps with decoding computation, completely eliminating the GPU memory overhead of auxiliary states.

DualDecoder implements a dual-token decoding pipeline to accurately identify critical KV entries with negligible computational overhead. It also features a layer-aware transfer schedule to overlap prefetching with model computation and a layer-scoped memory manager to reduce GPU runtime buffer. Experimental results demonstrate a decoding throughput improvement of up to 2.62x over state-of-the-art systems, with no degradation in latency or model quality. This breakthrough makes serving agentic applications with long contexts much more efficient.