Neural retrievers favor mainstream summaries, penalize niche content

A new paper from Valentini, Altszyler, and Fajcik reveals that neural retrievers don't just learn relevance—they learn implicit preferences for certain document types. The researchers trained classifiers on frozen document embeddings from three state-of-the-art supervised dense retrievers and found consistent evidence of a query-independent relevance prior. This prior creates a "findability gap": documents with lower prior scores are harder to retrieve even when they are genuinely relevant to the query. The effect is strong and persistent across multiple IR benchmarks and is less pronounced in the traditional BM25 model.

Using LLM-based explanations, the team identified what kinds of documents are favored: comprehensive, self-contained summaries of mainstream topics. In contrast, niche, fragmentary, or highly technical content tends to be under-represented in training annotations, and the retrievers internalize this bias. The study controlled for matched-document comparisons to confirm the effect is not due to query-document mismatch.

This work exposes a structural limitation of supervised retrieval: models trained on annotated data learn not only genuine relevance signals but also the selection biases in their training data. As neural retrieval becomes the backbone of search engines, RAG pipelines, and AI agents, these hidden preferences could systematically disadvantage specialized or less mainstream content, reducing the diversity of retrieved information and potentially reinforcing echo chambers.