New AI Dataset Teaches Computers to Peer Review Science

Researchers just built a massive collection of real scientific peer reviews to teach AI how to judge research papers. Called FIRSTPASS, it includes 3,668 complete review sessions from the journal Nature Communications, covering fields like biology, chemistry, and neuroscience. Each review shows the back-and-forth between reviewers and authors, including the editor's final decision.

Until now, AI training data only included short reviews from computer science conferences. But real science journals have much longer, more detailed critiques — averaging 2,155 words each. This new dataset captures that complexity, including requests for lab data, questions about methods, and debates over interpretations. It’s like giving AI a textbook example of how real science gets vetted.

Why does this matter? Better AI peer reviewers could speed up science by catching mistakes early, reducing wasteful retraction. It might also help researchers get clearer feedback on their work — or even help journals handle the flood of submissions. The team released everything for free, so any scientist or AI developer can use it.

There’s a catch: AI trained on this data would only be as good as the human reviewers it learns from. And real peer review involves judgment calls that can be subjective. Still, this is a big step toward making AI a useful assistant in scientific publishing.