Scientists Are Letting AI Write Their Code — and Rarely Double-Check It
Researchers run AI-written code without testing it, a new study of 527 people finds.
AI that writes computer code has quietly become a standard tool in science labs. To find out how it is actually being used, researchers Gabrielle O'Brien, Reed Milewicz and Nasir Eisty collected 527 written answers from a 2025 survey of researchers who write code, most of them at U.S. universities. Each person described one real task, how they used an AI tool for it, and how they checked whether the result was right. The team then sorted those answers by task type, experience level, and how confident people felt.
The work clustered into five jobs: handling data, making visualizations, debugging, mathematical and scientific computing, and statistical analysis. The checking was where things got shaky. More than half of the accounts described simply running the AI-written code and seeing what happened. Automated tests, which catch mistakes a person might miss, were rare. Having a colleague review the code was also rare. In other words, the safety net was usually one person's judgment, at their own desk, with no shared process behind it.
The most striking finding was about confidence. People with less programming experience trusted the AI more than they trusted themselves. Experienced programmers felt the opposite. Yet how confident someone felt about their own checking had almost nothing to do with what checking they actually did. It tracked two things instead: how much they trusted the tool, and how much they trusted themselves. Feeling careful and being careful, it turns out, are different things.
This matters because scientific code often sits underneath published results, medical models, climate projections, and policy decisions. A quiet bug can travel far. The authors suggest a fix that is less about blaming users and more about design: AI coding tools could prompt people to verify results in ways that fit the task at hand, instead of leaving every judgment call to the individual.
- 527 researchers, mostly at U.S. universities, described how they use AI to write code, and the study sorted their answers into five common jobs like cleaning data, making charts, and running statistics.
- Over half checked the AI's work by just running the code, while automated tests and review by another person were rare — leaving accuracy to individual judgment.
- Less experienced programmers trusted the AI more than themselves, while experienced programmers trusted themselves more, and feeling confident did not mean someone checked more carefully.
Why It Matters
If AI-written science code goes unchecked, errors can slip into published research — and into decisions based on it.