AI agents ace individual tasks but fail end-to-end science pipeline

A new arXiv preprint (Horstmann et al., June 2026) presents an empirical evaluation of general-purpose coding agents on a real neuroscience pipeline for fly optogenetics. The researchers assessed agents on tasks significantly larger than existing benchmarks, using datasets orders of magnitude bigger and evaluation criteria grounded in domain expert standards. Agents could solve several individual stages of the pipeline, suggesting that stage-level automation is tractable and could save scientists weeks of work. However, when it came to connecting all stages end-to-end, agents consistently failed.

The study reveals that the core challenge is scientific judgment. Agents struggle most when they must self-assess their own solutions without a predefined criterion to iterate on. They occasionally attempt visual inspection of intermediate outputs—mirroring human scientific practice—but largely fail to interpret what they see or take appropriate corrective actions. Additional obstacles include managing computational resources effectively and generalizing to large, held-out data collections that differ from training distributions. The authors distill principles for constructing scientific tasks and rigorous evaluation criteria for open-ended problems, emphasizing that current benchmarks miss these crucial aspects of real-world research.