KV-PRM slashes multi-agent LLM scoring 5000x using cache reuse
Process reward modeling just got 5000x cheaper by reusing KV caches.
Process Reward Models (PRMs) are critical for guiding test-time scaling in LLM-based multi-agent systems, but existing text-based PRMs re-encode entire trajectory text from scratch. This creates a quadratic O(LĀ²) bottleneck that grows with sequence length L, making them impractical for long-context multi-agent rollouts. KV-PRM solves this by directly reading the KV cache produced during the LLM's generation phase. Instead of full re-encoding, it processes a single 'verify token' against the pre-existing KV cache, reducing scoring cost to O(L). The authors formally prove that KV cache contains strictly more information than text for reward modeling.
Empirically, KV-PRM matches or exceeds text-PRMs on MATH, GSM8K, and AIME under Beam Search, MCTS, and Weighted Voting. The efficiency gains are dramatic: 5,000x fewer FLOPs, 37x lower latency, and 34x less memory per sequence. This breakthrough enables scalable, real-time process reward modeling for long-horizon multi-agent reasoning without sacrificing accuracy. The paper is submitted to arXiv (2607.09153) and represents a significant step in making test-time scaling economically viable for production systems.
- KV-PRM reduces scoring cost from O(LĀ²) to O(L) by reusing KV caches from LLM generation.
- Achieves up to 5,000x reduction in scoring FLOPs, 37x latency reduction, and 34x memory savings versus text-PRMs.
- Matches or outperforms text-PRMs on MATH, GSM8K, and AIME across three test-time scaling methods.
Why It Matters
Enables scalable, efficient process reward modeling for long-context multi-agent reasoning at fraction of cost.