UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval

A team of researchers, including Tobias Schimanski and Markus Leippold from the University of Zurich, has published a paper introducing UsefulBench, a novel dataset designed to challenge how AI systems retrieve information. The core problem is that conventional information retrieval (IR) and modern retrieval-augmented generation (RAG) systems are optimized for finding text that is semantically similar to a query, not text that is practically useful for making a decision. For example, when asking "Is Paris larger than Berlin?", a system might retrieve a text stating "Paris is in France" because it shares keywords, even though it contains no comparative population data. UsefulBench addresses this by providing expert-labeled data that distinguishes mere relevance from actionable usefulness.

The researchers curated the dataset using three professional analysts who labeled whether texts were connected to a query (relevance) or held practical value for answering it (usefulness). Their analysis revealed that classic similarity-based IR methods have a strong bias toward relevance over usefulness. While LLM-based systems can partially counteract this bias, the study found that domain-specific problems require a high degree of expertise that current models like GPT-4 do not fully incorporate. The paper concludes that UsefulBench presents a significant challenge for the next generation of targeted IR and RAG systems, pushing developers to move beyond semantic similarity as the primary goal.