A Systematic Study of Biomedical Retrieval Pipeline Trade-offs in Performance and Efficiency
New research benchmarks retrieval pipelines across 5 query types for biomedical NLP.
A new paper from Hayk Stepanyan and Matthew McDermott provides concrete guidance for building biomedical retrieval systems, addressing a critical gap in practical advice. The study systematically evaluates how pipeline design choices—corpus selection, chunk granularity, and vector index configuration—affect performance and efficiency at scale. They tested over diverse query types, including exam-style questions, conversational medical queries, community-asked questions, and non-question formulations, using public biomedical text datasets.
Key findings include the superiority of corpus aggregation for absolute retrieval quality, and the identification of MedRAG/pubmed as the Pareto-optimal singleton corpus under graph-based HNSW indexing. The authors also recommend appropriate chunking strategies and FAISS indexing choices that offer the best trade-offs in speed and efficiency. All results were validated using a robust LLM-as-a-judge assessment with human validation.
- Corpus aggregation significantly outperforms singleton corpora for absolute retrieval quality in biomedical pipelines.
- MedRAG/pubmed is identified as the Pareto-optimal singleton corpus under HNSW indexing, balancing performance and efficiency.
- FAISS indexing with appropriate chunking strategies provides the best speed-efficiency trade-offs across diverse query types.
Why It Matters
This study offers actionable benchmarks for building efficient biomedical retrieval systems, critical for clinical NLP and AI-assisted diagnosis.