New protocol uses RAG and majority voting to detect ChatGPT hallucinations in biomedicine

A new study from researchers Ahmed Abdeen Hamed and Luis M. Rocha introduces a protocol to systematically evaluate ChatGPT's ability to generate and verify biomedical associations centered on diseases. The paper, published in STAR Protocols 2026, addresses a critical need in AI-driven biomedical research: how to trust associations produced by large language models (LLMs) that are prone to hallucinations—plausible but incorrect statements. The protocol first asks ChatGPT to generate disease-centric associations from unstructured text, then validates the biological entities against standard biomedical ontologies. A self-consistency strategy assesses how reliably different ChatGPT models produce the same associations, flagging inconsistencies as potential hallucinations.

To overcome limitations of exact ontology matching, the protocol adds a semantic verification step using Retrieval-Augmented Generation (RAG) powered by open-source LLMs. This RAG workflow retrieves relevant literature from curated databases and uses a cross-model majority voting mechanism: multiple open-source LLMs vote on whether an association is supported by the retrieved evidence. If the majority rejects the association, it is flagged as likely hallucinated. This approach creates a decentralized truth-establishment system where one LLM's output is verified by others, reducing reliance on any single model. The authors provide code and supplementary data to replicate the protocol, aiming to give researchers a robust tool for validating AI-generated biomedical knowledge before using it in clinical or drug discovery workflows.