CacheCraft discovers KV eviction policies, boosting RULER scores by 15 points
LLM-guided evolution finds a compact scorer that beats hand-crafted heuristics by +67 points.
KV cache compression is critical for long-context inference, but designing eviction policies has remained a manual, brittle process. Existing prefill-stage methods rely on hand-crafted salience heuristics that often fail across different models, context lengths, and compression ratios. Researchers Pratik Poudel, Yanzhao Wu, Sumit Jha, and Jason Liu present CacheCraft, a program-evolution methodology that uses an LLM-guided code-evolution engine to automatically discover effective eviction policies.
CacheCraft discovered FRC (Feature-Rich Compression), a fixed-weight three-signal scorer that combines local attention received, neighborhood attention density, and KV-head maximum salience with chunk-level top-k selection. Without any per-model retuning, FRC ranks first among evaluated single-pass baselines at every RULER 4k/8k cell with r >= 0.75 across Llama-3.1-8B-Instruct and Qwen3-8B (12 of 20 grid cells), gaining +15.4 points on Llama-4k and +13.9 points on Qwen-8k at 88% compression. A decomposition analysis reveals the scoring family is the load-bearing design choice: incorporating the scorer contributes +67.2 RULER points, while improving chunk structure contributes only ~0.1. Beyond FRC itself, CacheCraft provides a transferable recipe for automated eviction-policy discovery, including a compact policy interface, a cascade evaluator with strict output invariants, and a diagnostic loop that treats plateaus as evidence for reformulating the editable interface.
- CacheCraft uses LLM-guided program evolution to search KV cache eviction policies, discovering FRC automatically.
- FRC scores +15.4 on Llama-4k and +13.9 on Qwen-8k at 88% compression, beating all single-pass baselines in 12/20 RULER cells.
- Scorer design drives +67.2 RULER points, while chunk structure only adds ~0.1, showing scoring is the key lever.
Why It Matters
Automated eviction-policy discovery could replace brittle hand-crafted heuristics, enabling efficient long-context inference across models without retuning.