Test Shows AI Can Sort Scientific Papers But Misses Key Details

Scientists often need to review every published study on a topic to find reliable evidence—think medical guidelines, new drug research, or climate policy. This process, called a systematic review, usually means reading tens of thousands of papers. Now, AI tools like ChatGPT promise to help. But how well do they actually work? A new study introduces SciLitBench, a way to test AI systems across the entire review process, rather than just one small step.

The benchmark uses over 42,000 paper records and nearly 900 fully annotated studies to test 22 different AI models. Researchers found that AI does remarkably well at the first stage: sorting which papers are worth reading. When given clear inclusion and exclusion rules, the best models improved by nearly 29% at finding relevant studies. Adding short explanations written by human researchers also boosted full-text screening by 15%. This suggests AI could take over the most tedious part of a researcher's job—the initial triage of thousands of papers.

However, the test reveals a sharp drop-off when AI has to extract specific facts from a finished paper. For simple data like publication year, AI was 97% accurate. But for more complex information, such as describing a paper's computational method, accuracy fell to just 37%. Even the strongest AI systems retrieved only 30% of the evaluation evidence and 25% of the limitations that human experts identified. In other words, AI may skim papers well, but it can't yet be trusted to fully understand or summarize the nuances that matter for critical decisions.

The takeaway is practical: AI can dramatically reduce the time researchers spend searching and screening, letting them focus on the harder job of interpreting evidence. But relying on AI for detailed data extraction could lead to incomplete or misleading conclusions. SciLitBench gives researchers and developers a shared way to measure these limits. As AI keeps improving, tools like this will be essential to ensure that what works in the lab actually helps in the real world—without introducing unseen errors into science.