[R] ARC Round 3 - released + technical report

The ARC Prize, a benchmark designed to measure true abstraction and reasoning capabilities akin to human fluid intelligence, has published its Round 3 results and a revealing technical report. The findings are stark: every frontier AI model tested—including offerings from OpenAI, Anthropic, and Google—failed to surpass a 1% score on the novel visual puzzle tasks. This indicates that despite massive scale and training data, current systems lack core, efficient reasoning abilities. The report further suggests, through analysis of model reasoning traces, that the few models performing relatively well likely had exposure to ARC-like data during training, rather than developing the skill from first principles.

The continued failure to claim the substantial prize money from Rounds 1 and 2 underscores the benchmark's difficulty and the field's current limitations. The ARC-AGI tasks require solving unique visual pattern problems that cannot be memorized, demanding genuine abstraction. The sub-1% scores across the board reveal a vast "room for improvement" in developing AI that can reason efficiently and generalize on unseen problems. This result challenges the narrative of rapid, continuous improvement in AI reasoning and highlights a specific frontier where even the most advanced models fall short, pointing researchers toward the hard problem of building systems that learn to learn.