Are LLM-Based Retrievers Worth Their Cost? An Empirical Study of Efficiency, Robustness, and Reasoning Overhead

A comprehensive study accepted at SIGIR 2026, led by researchers including Abdelrahman Abdallah, rigorously evaluates the practical value of LLM-based retrievers for complex queries. The team reproduced the BRIGHT reasoning-intensive benchmark across 12 tasks, testing 14 different retrievers. They extended traditional accuracy metrics to include critical operational factors: cold-start indexing cost, query latency distributions, throughput under load, corpus scaling behavior, and robustness to query perturbations. A key finding is that some specialized retrievers under 1 billion parameters achieve competitive effectiveness while maintaining high throughput, whereas several large LLM-based bi-encoders impose significant latency penalties for only incremental accuracy improvements.

The research introduces and quantifies 'reasoning overhead' by comparing standard queries to five reasoning-augmented variants, measuring the accuracy gain per unit of added latency. This analysis shows reasoning augmentation adds minimal latency for efficient sub-1B encoders but yields diminishing returns for top-tier models and can even hurt performance in formal domains like math and code. Perhaps most critically, the study finds confidence calibration is consistently weak across all model families; the raw retrieval scores are unreliable for downstream decision-making like query routing without additional, costly calibration steps. The authors have released all code and artifacts, providing a vital resource for engineers building production RAG systems who must balance performance with real-world cost and latency constraints.