AI Still Loses to Humans at Summarizing Research — But It's Getting Closer

Writing a literature review — a summary of all the existing research on a topic — is one of the most tedious parts of being a scientist. It can take days or weeks of reading. AI could do it much faster, but how do you know if the AI is actually good? That's the problem these researchers set out to solve.

Their idea was to use a "battle platform." Human experts with experience in AI-assisted writing compared anonymous drafts — some written by humans, some by AI — and picked the better one. Think of it like a tournament where every match is judged by a specialist. After nearly 3,000 expert judgments, they found that even the strongest AI systems won only 23% of decisive matches against human-written reviews. However, "agentic" AI systems (AI that can search databases and reason through steps) performed 60% better than basic language chatbots, showing that smarter AI is making real progress.

The catch is that when they used AI to judge other AI, those automated judges disagreed significantly with actual human experts — they only agreed about half the time. That means you couldn't trust an AI to grade its own work. So the team built a new "expert-calibrated" evaluator called LitJudge, which improved agreement to nearly the same level as two human experts agreeing with each other. This makes automated evaluation far more reliable.

The bottom line? AI isn't ready to replace human researchers' judgment when it comes to summarizing science. But with tools like LitJudge, scientists can now responsibly use AI to speed up the grunt work while keeping quality checks human. The team published their code and data online, meaning this isn't just a theory — it's a practical tool the research community can start using today.