Hybrid retrieval boosts public health QA accuracy in new RAG study

A new study by Felix Feldman and colleagues tackles a critical problem: how to make large language models (LLMs) reliable for public health question answering, where hallucinations and rapidly evolving guidance are major risks. The team extended PubHealthBench, a benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented generation (RAG) setting. They systematically compared dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants. Their key technical finding: hybrid retrieval consistently outperformed single-method approaches in recall and ranking quality, though chunk length and topic interact with performance.

Beyond retrieval, the study evaluated how context provision impacts model accuracy. Providing retrieved context significantly boosted multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or beat larger models that lacked retrieval support. To assess realistic free-form answers, the authors introduced a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency. They validated this against dual human annotations, finding strongest agreement on faithfulness and completeness, but weaker reliability for factual consistency and clarity. Overall, the results emphasize that retrieval quality is the primary lever for reliable public health QA, offering practical guidance for building RAG systems grounded in official guidance.