VarRate slashes LLM memory 80% without retraining

The key-value (KV) cache is the main memory bottleneck during long-context LLM inference. Existing training-free approaches fall into two flawed families: token-selection methods (SnapKV, Ada-KV) irreversibly evict low-scoring tokens, causing accuracy collapses of 11-15 points when importance signals degrade; uniform low-rank coding wastes budget by spending equal rank on every token. VarRate solves both by allocating rank rather than evicting tokens. It is a training-free KV codec that assigns each token a variable low-rank budget proportional to its query salience, keeping every token at a nonzero rank.

Tested on the 16-task LongBench at a 20% compression budget, VarRate degraded by only 0.8 points from the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. It significantly outperformed its uniform-rank ablation and was accuracy-equivalent to KVzip (a query-agnostic reuse method) in three of four settings, while requiring roughly one-eighth the prefill overhead. This makes VarRate the strongest matched-memory compressor among training-free methods, enabling longer context processing without costly retraining.