New AI method slashes LLM memory use by 50% with smarter caching

Researchers introduced KV Policy (KVP), a reinforcement learning framework that optimizes the memory-intensive Key-Value cache in Large Language Models. Instead of using simple heuristics, KVP trains lightweight agents to predict which cached tokens are most useful for future text generation. The method significantly outperforms existing baselines on long-context benchmarks like RULER and generalizes well to tasks like LongBench, enabling more efficient inference without modifying the core model architecture.