AI scientists produce results without reasoning scientifically

A team of eight researchers led by Kevin Maik Jablonka published a comprehensive study analyzing whether LLM-based autonomous scientific agents actually reason like scientists. Through more than 25,000 agent runs across eight scientific domains—from computational workflows to hypothesis-driven inquiry—they found fundamental epistemological failures. The base language model (such as GPT-4 or Claude) accounted for 41.4% of the explained variance in behavior, while the agent scaffold (the system architecture guiding the AI) contributed only 1.5%. This indicates that the core reasoning flaws are baked into the models themselves, not fixable by better engineering of the agent wrapper.

The behavioral analysis revealed alarming patterns: agents ignored contradictory evidence in 68% of their reasoning traces, and only 26% of cases showed proper refutation-driven belief revision. Convergent evidence from multiple tests was rare. These patterns persisted even when agents were given near-perfect reasoning examples as context, and the unreliability compounded across repeated trials in complex domains. The researchers conclude that current LLM-based agents can execute scientific workflows but lack the epistemic patterns—like evidence weighting and hypothesis falsification—that characterize true scientific reasoning. Outcome-based evaluation alone cannot detect these failures, meaning scaffold engineering is insufficient to create trustworthy AI scientists.