Kara's sliding-window KV cache slashes LLM serving costs by up to 50%

Researchers from an academic team introduce Kara, a novel KV cache compression technique designed to tackle the memory and latency bottlenecks of reasoning LLMs. During the decoding phase, long chain-of-thought (CoT) outputs accumulate massive key-value caches, often exceeding available GPU memory. Kara addresses this by applying sliding-window compression on the fly: instead of pruning the entire cached sequence, it scores only the recently generated tokens using bidirectional attention. Importantly, it avoids the rigid boundaries of fixed-size chunks by introducing a Token2Chunk module that groups selected KV pairs into adaptive, semantically meaningful blocks. This allows Kara to retain important information from arbitrary token positions without fully eliminating any sequence block.

Kara is built on top of vLLM via the KvLLM inference framework, adapting the sliding-window method to PagedAttention for practical serving. The result is a significant reduction in KV cache memory usage—up to 50% lower in experiments—while preserving model accuracy on reasoning benchmarks. Unlike previous compression methods that either provide limited throughput gains or risk information loss, Kara consistently improves output throughput across multiple LLM architectures. The paper includes 9 pages of evaluations on standard language and reasoning tasks, showing that reasoning quality remains intact even with aggressive compression. For production systems serving models with long reasoning traces, Kara offers a drop-in optimization that requires no retraining or model modification.