New KV Cache Filter Fixes LLM Accuracy Collapse on JSON

A new paper by Soumil Mandal diagnoses a critical flaw in attention-based KV cache eviction methods like H2O. When processing schema-dense inputs such as nested JSON, these techniques disproportionately retain structural tokens (delimiters, whitespace) over content-bearing tokens. The author shows that structural KEY tokens are retained at 1.8x the rate of answer-carrying VALUE tokens, causing exact-match accuracy to collapse from 88% to 0% at a 5% cache budget. The problem stems from the accumulated attention mass metric, which acts as a non-stationary filter that amplifies noise from non-content sink roles.

The proposed solution is a retraining-free, role-conditional allocation method built atop SnapKV's windowed score. A single tuned hyperparameter governs how cache budget is distributed across token roles, with a counterfactual experiment confirming that suppressing KEY tokens is the most effective strategy. Using a 15 MB linear role probe to supply role labels at negligible inference cost, the method closes 63-98% of the H2O accuracy gap at sub-20% budgets. At higher budgets, it modestly matches or exceeds full-cache accuracy, showing a small denoising effect. However, matching parser-level downstream accuracy remains an open challenge.