CORE-Bench study shows AI benchmarks need more than accuracy

A new paper from Princeton researchers (Nadgir et al.) challenges the reflex to retire saturated AI benchmarks. Focusing on CORE-Bench Hard—a benchmark for computational reproducibility of scientific code—they demonstrate that even after accuracy plateaus, agents can be meaningfully compared along six other axes: construct validity (e.g., shortcut exploitation), out-of-distribution generalizability, efficiency, reliability, the relative contribution of the model vs. its scaffold, and the uplift from human-agent collaboration. They surface threats to construct validity that weaker agents never exposed, leading to an improved CORE-Bench v1.1 and a new OOD task suite.

Concretely, the team found that despite accuracy saturation, CORE-Bench v1.1 remains useful for measuring efficiency and reliability, and can disentangle model performance from scaffold design. In a small-scale randomized experiment, human-AI collaboration achieved a statistically significant 2x speedup over humans alone—likely an underestimate since one-fifth of human-only runs hit the time limit. These results argue for a multi-dimensional evaluation ecosystem that extends the useful life of benchmarks and yields richer insights into agent capabilities.