ScholarQuest Benchmark Reveals a Shocking Gap: Why Your AI Search Agent Misses 69% of Academic Papers
Best agent scores just 0.314 Recall@100 on new 1,000-topic academic search test
A team led by Tingyue Pan (with co-authors from multiple universities) has introduced ScholarQuest, a large-scale benchmark specifically designed to evaluate how well LLM-based agents perform academic paper search in open literature environments. The benchmark covers over 1,000 computer science topics and defines four distinct research intents: method-oriented, setting-anchored, comparison-based, and scope-controlled queries. This taxonomy-guided approach allows for systematic assessment of agentic search capabilities beyond simple keyword matching.
Initial benchmarking results reveal that while LLM-based agentic methods outperform single-shot retrieval baselines, performance remains surprisingly low: the best agent achieves only 0.314 Recall@100 and 0.355 Recall@All. The study also analyzes search efficiency, intent-level robustness, and failure cases, providing multi-dimensional evaluation signals. ScholarQuest includes a shared retrieval backend called ScholarBase to ensure reproducible comparisons. The findings highlight substantial room for improvement in AI-powered academic search, especially for nuanced research intents that require iterative refinement.
- Covers over 1,000 computer science topics across four research intents (method-oriented, setting-anchored, comparison-based, scope-controlled).
- Top-performing agent achieves only 0.314 Recall@100 and 0.355 Recall@All, indicating major gaps in agentic search capabilities.
- Includes reproducible retrieval backend (ScholarBase) and failure case analysis for multi-dimensional evaluation.
Why It Matters
Sets a rigorous benchmark for improving AI-driven literature search, essential for accelerating scientific research with LLM agents.