HETERQA benchmark tests AI retrieval across 5 data sources, finds big gaps

HETERQA is a comprehensive benchmark designed to evaluate record retrieval when data comes from multiple heterogeneous sources—relational tables, text documents, image repositories, spatial databases, and knowledge graphs. Built using Yelp business records, the dataset contains 857 question-answering pairs that require models to jointly exploit these sources. The benchmark was constructed in an answer-driven manner: candidate records are filtered by constraints, enriched via heterogeneous sources, then cross-verified before the natural-language question is finalized. Human validation and contradiction detection ensure quality.

Initial evaluations using sparse, dense, hybrid, late-interaction, and agentic retrievers show that no method saturates the benchmark. Hybrid retrieval achieved the strongest Recall@10, while Self-RAG (retrieval-augmented generation with self-reflection) achieved the best MRR@10. These results indicate that current retrieval methods—including advanced agentic ones—are insufficient for real-world heterogeneous data scenarios. HETERQA thus provides a challenging testbed and leaves substantial room for future retrieval innovations. The dataset and code are publicly available.