Verification agent solves ARC-AGI-3 with OpenAI GPT-5.6-sol, 99% RHAE

The study by Sergey Rodionov (arXiv 2607.15439) investigates which components drive performance in coding agents on the ARC-AGI-3 challenge. Four nested variants were built: a textual baseline, a flexible-interface executable world model without replay verification, the same with scheduled simplification, and a fixed-interface verification treatment requiring exact reproduction of recorded observations. Evaluations used gpt-5.4, gpt-5.5 at high/xhigh, and gpt-5.6-sol at xhigh/max. Results show every variant improves with stronger models and more reasoning effort. Surprisingly, the textual baseline outperformed the executable world model in some settings, and simplification helped in most cases. The full verification treatment ranked first across all settings but used substantially more resources. In follow-ups with gpt-5.6-sol, the verification variant fully solved every public game at both reasoning efforts, achieving about 99% RHAE and using fewer than half the human's total actions. However, since gpt-5.6-sol postdates the games, this likely indicates saturation of the public test set rather than true AGI progress.

These findings highlight important nuances in agent design: executable world models are not universally beneficial, and simple textual reasoning can sometimes outperform when paired with strong base models. The verification step—exact replay of recorded observations—provides a robust safety net but at significant computational cost. For practitioners, the key takeaway is that investing in stronger base models and higher reasoning effort yields consistent gains, while the marginal benefit of architectural complexity (executable models, simplification) varies. The paper also warns against overinterpreting public benchmark saturation as genuine capability improvement. With held-out performance untested, the results serve as a cautionary tale for benchmark-driven AI research.