VisDocAgentBench reveals agentic retrieval beats static 27x on complex docs

VisDocAgentBench, from researchers including Lexiang Hu, Yanzhao Zhang, and Zhouchen Lin, is a new benchmark that systematically compares static and agentic retrieval on visually rich documents under a shared ranked-output contract. Unlike prior benchmarks that rely on question answering or report generation, this work directly evaluates document ranking under iterative evidence acquisition. The corpus contains 2,375 pages from 100 documents with 120 unique-target queries designed to vary evidence complexity: direct, one-bridge (requiring one intermediate document link), and two-bridge (requiring two). Relations between documents are preserved to generate semantic, relational, and visual queries, then validated with hard negatives and full-document review.

The results expose a sharp gap in retrieval capability. A strong late-interaction visual retriever achieves 97.50% Recall@1 on direct items but plunges to 2.50% on two-bridge items, proving that one-shot query–page matching fails when relevance depends on corpus context. Agentic retrieval via iteratively searching and inspecting pages recovers much of this loss, but choices matter: every planner performs significantly better with visual retrieval (best 67.50% R@1) than with OCR-text (37.50% R@1). Ablations highlight iterative search and page inspection as the most consequential agent capabilities, and providing full support context improves ranking on both routes. Trace analysis localizes remaining failures to target discovery, candidate examination, and evidence-role integration. The authors argue retrieval agents must combine modality-preserving discovery with evidence-directed verification, making VisDocAgentBench a practical guide for building agentic RAG systems.