Hybrid Retrieval for COVID-19 Literature: Comparing Rank Fusion and Projection Fusion with Diversity Reranking

Researcher Harishkumar Prajapati has developed and evaluated a sophisticated hybrid retrieval system specifically designed for navigating the massive corpus of COVID-19 scientific literature. The system, tested on the established TREC-COVID benchmark containing 171,332 papers and 50 expert queries, implements six different configurations that strategically combine sparse retrieval (using the SPLADE model) and dense retrieval (using the BGE model). The core innovation lies in comparing two fusion techniques: traditional rank-level fusion (RRF) and a novel projection-based vector fusion method called B5. The RRF approach achieved the highest relevance score, with an nDCG@10 of 0.828, outperforming dense-only retrieval by 6.1% and sparse-only by 14.9%.

While RRF led on pure relevance, the projection fusion variant (B5) offered compelling trade-offs, being 33% faster (847 ms vs. 1271 ms per query) and generating results with 2.2 times higher intra-list diversity (ILD@10). The evaluation expanded to 400 queries, including expert, machine-generated, and paraphrased styles, revealing that B5 delivered its largest relative gains (+8.8%) on keyword-heavy reformulations. The research also explored diversity reranking with MMR, which increased result diversity by 23.8-24.5% at a cost of 20.4-25.4% in relevance score. Critically, both fusion pipelines maintained sub-2-second latency targets. The complete system is deployed as a functional Streamlit web application, utilizing Pinecone's serverless vector indices for scalable backend operations.