Entity-based retrieval fails to beat BM25 in open-world tests, study finds

Shubham Chatterjee's arXiv paper "Entities as Retrieval Signals" systematically investigates why entity-oriented retrieval methods often report conflicting results. On TREC Robust04, the study evaluated 6 neural rerankers and 437 unsupervised configurations using entity signals against the BM25 baseline. Across all 443 systems, none improved Mean Average Precision (MAP) by more than 0.05 when evaluated over the full candidate set (open-world), despite strong gains in entity-restricted (conditional) settings. The best configuration matched the official Robust04 best system and outperformed most neural rerankers, showing that architecture is not the limiting factor—the entity channel itself is.

The root cause is coverage: even under idealized entity selection, signals cover only 19.7% of relevant documents. The paper introduces a key distinction between Conceptual Entity Relevance (CER)—semantic relatedness—and Observable Entity Relevance (OER)—corpus-grounded discriminativeness under a given linker. All supervision strategies operate at the CER level and ignore the linking environment, producing signals that are semantically valid but not discriminative in practice. Improving supervision actually reduces coverage without improving effectiveness. The study concludes that conditional and open-world evaluations answer different questions, and progress requires dataset-level entity discriminativeness plus metrics reporting both coverage and effectiveness.