Project Auto-World uses LLMs to automate AI reasoning benchmarks

Project Auto-World, a new paper from researchers including Anirvan Das and others, tackles the challenge of evaluating neural models that reason about relational structures. These models often struggle to generalize to harder instances than those seen in training, and it's unclear what makes an instance hard. The team proposes using large language models (LLMs) to automate the generation of benchmarks. Specifically, given a world parameterized by Datalog rules and an Edge Transformer as the reasoning evaluator, they use LLM-driven evolutionary search (based on Google DeepMind's FunSearch) and autonomous agentic search to discover sampling functions that produce increasingly difficult problem instances.

The results show that the Edge Transformer can be improved using this auto-generated data, generalizing well to further data perturbations. Crucially, the same machinery can be applied to novel worlds proposed by LLMs, enabling fully autonomous research on neural relational reasoning. This approach sidesteps the traditional bottleneck of manually designing challenging benchmarks, potentially accelerating progress in AI's ability to handle structured reasoning. The work has been submitted to the NeurIPS 2026 Evaluation & Dataset track, with code available on GitHub.