ICML 2026 paper warns AI agents risk science's trustworthiness
Autonomous research agents create verification gaps that threaten scientific integrity...
Belinda Mo's paper, "The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science," accepted at ICML 2026's Position Paper Track, tackles a growing crisis: AI systems operating as autonomous research agents. These agents generate hypotheses, design experiments, and produce discoveries at scales that outpace human oversight. The paper notes that current submission trends in machine learning venues already show a widening verification gap—our ability to check scientific output lags behind its production. Autonomous agents worsen this by magnitudes due to human-agent asymmetry: agents can't be questioned or sanctioned like human contributors.
The author argues that science must evolve its verification infrastructure, just as it did with peer review. However, past adaptations assumed human actors who could be held accountable—AI agents break that assumption. Mo proposes criteria for an adapted infrastructure: observable-by-default workflows, scalable verification methods, and clear attribution. Without adaptation, ML and any domain using agents risk dangerous failures—experimental results no person can verify, optimization for metrics over understanding, and accountability vacuums that erode scientific trust.
- AI agents now autonomously generate hypotheses, design experiments, and produce discoveries beyond human oversight, widening the verification gap.
- The paper proposes three criteria for a new verification infrastructure: observable workflows, scalable verification, and clear attribution.
- Without adaptation, the system faces unverifiable experiments, metric optimization, and accountability vacuums that erode trust.
Why It Matters
As AI agents become researchers, we need new verification systems to keep science trustworthy and accountable.