Biomedical RAG study finds retrieval barely improves accuracy
Across 10 datasets and 5 models, RAG added just 1-2 points.
A comprehensive study, accepted at BioNLP Workshop at ACL 2026, systematically re-evaluates the effectiveness of retrieval-augmented generation (RAG) for biomedical question answering. The authors tested 5 open-weight instruction-tuned models (from 7B to 72B parameters) across 10 diverse biomedical QA datasets, using 4 different retrieval methods and 4 distinct retrieval corpora (including both expert-level and layman sources). The surprising finding: RAG yielded only small and inconsistent improvements, typically within 1-2 accuracy points over a baseline without any retrieval.
More importantly, the choice of the backbone model had a much larger impact on performance than any combination of retriever or corpus, and expert and layman retrieval sources performed similarly in most settings. This suggests that current models—even large ones—struggle to effectively integrate retrieved evidence, pointing to a fundamental limitation in instruction-following and context utilization rather than retrieval quality. The study challenges the prevailing assumption that RAG is a reliable way to improve factual accuracy in high-stakes medical QA and shifts focus back to improving the underlying model's reasoning capabilities.
- Across 5 models (7B-72B), 10 datasets, 4 retrievers, and 4 corpora, RAG improved accuracy only 1-2% over no-retrieval baselines.
- Backbone model choice had a much larger effect on performance than the choice of retriever or retrieval corpus.
- Expert and layman retrieval sources performed similarly, indicating models fail to leverage retrieved evidence effectively.
Why It Matters
Challenges the assumption that RAG is a silver bullet for medical QA, shifting focus to model reasoning improvements.