iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics
New benchmark shows LLMs struggle with real-world questions requiring evidence synthesis from multiple sources, not just retrieval.
A team of researchers from the University of Washington and Rutgers University has introduced iAgentBench, a new benchmark designed to rigorously test the 'sensemaking' capabilities of AI agents that search and synthesize information. The core problem it addresses is that most current Question-Answering (QA) benchmarks are too simplistic, often answerable by retrieving a single relevant passage. In contrast, iAgentBench constructs questions based on real-world, high-traffic topics and common user intent patterns, forcing AI systems to perform higher-level tasks like integrating conflicting evidence, tracking causal links, and resolving dependencies across multiple sources. Each benchmark instance includes traceable evidence and auditable intermediate artifacts, enabling detailed failure analysis and contamination checks.
Initial experiments with multiple Large Language Models (LLMs) reveal a significant shortcoming in today's AI: while access to retrieval tools (like web search) improves answer accuracy, retrieval alone is insufficient to reliably solve these complex, multi-faceted questions. This underscores a critical distinction between mere evidence *access* and genuine evidence *use*. The benchmark's release signals a shift in AI evaluation toward more realistic, challenging tasks that mirror how professionals and the public actually seek information online. It provides a necessary tool for developers to diagnose whether their agents fail at finding the right information or at the crucial synthesis step, pushing the field toward building AI that can truly reason across documents.
- Benchmarks 'sensemaking'—the ability to integrate and reconcile evidence from multiple sources, not just retrieve facts.
- Questions are grounded in real-world, high-traffic topics and user intent patterns, making them more realistic than standard QA tests.
- Experiments show retrieval improves LLM performance but is not a reliable solution, highlighting a gap in synthesis capabilities.
Why It Matters
It pushes AI beyond simple fact retrieval toward the complex, multi-source reasoning required for professional research and decision-making.