Auto Research for Materials: AI agent achieves 26.3% improvement on held-out data

The paper introduces a closed-loop AI research agent designed for materials science that goes beyond typical optimization. Instead of only maximizing a validation score, the agent is tested on a stricter criterion: do the selected changes survive on data that never entered the loop, and can the code be reused? The search is partitioned into changes to features, models, representations, and training data. Seven searches produce 701 evaluated changes across ten Matbench endpoints, with agents receiving only the mean over five inner folds to reduce overfitting. After freezing the selected code and evaluating on untouched holdout data, nine of ten choices remain the best single intervention.

The surviving changes reveal two distinct regimes: for composition-only tasks, feature, model, and representation changes offer comparable improvements (e.g., held-out MAE reductions of 17.4% for band gap and 18.6% for steel strength), while screened external data adds little. For structure tasks, richer geometry descriptors and model/calibration changes lower mean held-out MAE by 14.6% and 7.1%, whereas composition embeddings do not transfer. Combining separately found feature and model changes yields a 26.3% mean held-out improvement. These results demonstrate that auditable AI-scientist workflows can produce decisions that survive unseen evidence and create reusable code changes across tasks, establishing a robust evaluation design for executable discoveries beyond the feedback loop.