Study: Popular KV-Cache Methods Fail Under Query-Agnostic Compression

KV-cache compression is critical for scaling LLM inference, but existing evaluations almost always assume the query is visible before compression—a protocol that doesn't reflect real-world reuse scenarios where a compressed cache is queried multiple times. A new paper by Luo, Liang, and Xuan systematically audits six published compression methods (SnapKV, KeyDiff, etc.) against three trivial baselines under a matched-budget setup. They fix every variable—model (three 7-9B open models), compression ratio, instances, decoding—and only change the scoring rule to either query-aware or query-agnostic. With 144,300 paired evaluations on RULER-8192 and 40,800 on LongBench, they produce statistically rigorous results.

The findings are stark: under query-agnostic compression, only KeyDiff consistently beats a best-of-3 trivial baseline (31 of 36 cells). SnapKV, the most widely deployed method, loses to the simple 'keep start and recent window' baseline by an average of -0.066. The performance drop between protocols correlates directly with how much the query influences each method's scoring signal—SnapKV drops by Delta=+0.198 (its 64-token observation window includes the question), while KeyDiff drops only Delta=+0.011 (its score contains no query term). This suggests that many compression methods are overfit to unrealistic evaluation protocols and may not deliver savings in production systems where pre-computed caches are reused across many queries.