New AI Test Could Speed Up the Science Behind Your Medicine

Every trusted medical guideline — from how often to get a mammogram to which blood pressure pill works best — starts with a "systematic review." That means researchers gather every relevant study ever published on a question, read them, and decide which ones count. Sounds simple. It isn't. A single review can involve scanning tens of thousands of paper titles and abstracts, and the sorting is done largely by hand, often by a small team of graduate students. It can take months.

This new paper, from researchers at the University of Montreal and collaborators, offers a way to check whether AI chatbots (specifically large language models, the tech behind ChatGPT) can take over part of that sorting. The team built a test set of 45,064 labeled articles drawn from 32 real published reviews. Each entry is already marked "keep" or "skip," so an AI can be scored against a known answer key.

The clever part is a fairness fix. In real reviews, the overwhelming majority of papers get rejected — sometimes 95 out of 100. A lazy AI that just says "skip everything" would look accurate on paper while being useless. So the team designed their scoring to account for that imbalance, catching AI that games the numbers. They also released a free tool, PromptSR, that lets researchers test different instructions and track results.

The catch: this is a measuring stick, not a finished product. Nobody is suggesting AI replaces expert judgment. A model that wrongly skips one important study could quietly distort medical advice for years — and the paper itself exists precisely because current evaluation methods aren't trustworthy enough yet. Think of it as building a decent scale before you start weighing things.