IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering

A research team led by Connor Shorten has released IRPAPERS, a comprehensive benchmark for evaluating visual document processing systems in scientific contexts. The benchmark contains 3,230 pages from 166 scientific papers, each available as both an image and OCR transcription, with 180 carefully constructed 'needle-in-the-haystack' questions for testing retrieval and question-answering capabilities.

The study reveals that while text-based retrieval using Arctic 2.0 embeddings and BM25 achieved 46% Recall@1, and image-based retrieval reached 43%, combining both modalities through hybrid search boosted performance to 49% Recall@1. The most impressive result came from Cohere's Embed v4 model, which achieved 58% Recall@1 using page image embeddings alone, outperforming Voyage 3 Large text embeddings and all open-source models tested. For question answering, text-based RAG systems showed higher ground-truth alignment (0.82 vs 0.71), but both modalities benefited significantly from increased retrieval depth.

This research matters because visual documents—PDFs, scanned papers, and formatted reports—represent a massive portion of enterprise and scientific knowledge that traditional text-only systems struggle to process effectively. The benchmark demonstrates that multimodal approaches aren't just theoretical improvements but deliver measurable gains in recall accuracy. As organizations increasingly rely on RAG systems for knowledge management, IRPAPERS provides concrete evidence that investing in multimodal capabilities pays dividends, particularly for complex domains like scientific research where documents contain crucial visual information alongside text.