AV-AIVAT cuts agent evaluation cost 74x with anytime-valid stopping

Evaluating which AI agent is stronger typically requires playing thousands of games—each costing money in inference or expert time. Fixed-budget runs either overspend or stop too early, while naive optional stopping breaks statistical guarantees. Now researchers from Tsinghua and the University of Chicago have introduced AV-AIVAT (Anytime-Valid AIVAT), a method that combines the Action-Informed Value Assessment Tool (AIVAT) with continuously monitored Confidence Sequences (CSs) to stop evaluation the moment evidence suffices, without invalidating the confidence level.

In tests across 15 LLM agent configurations and 71,439 paired Heads-Up No-Limit Hold'em hands, AIVAT alone delivered a median 54x variance reduction. When paired with an Asymptotic CS at a 95% confidence level and ±1 Big Blind precision, AV-AIVAT required a median 74x fewer hands than raw outcomes to reach a confident verdict. For exact finite-sample certification, the team uses an Empirical-Bernstein CS (EB-CS), which they validated structurally for Leduc hold'em. The method cleanly separates asymptotic screening from exact certification, so evaluators can stop near-optimally and hand a third party all data needed to recheck the verdict at that exact stopping time.

The practical impact is significant for anyone benchmarking AI agents in imperfect-information settings—from poker AIs to LLM agents negotiating or playing multiplayer games. Instead of burning compute to ensure reliability, teams can now get certified answers faster and cheaper, while retaining full auditability of the stopping decision.