OCR pipelines beat vision LLMs on long document QA in new benchmark
The latest benchmark reveals that a classic OCR pipeline not only beats vision-capable LLMs on accuracy for dense document QA but also does so at a lower cost—a counterintuitive result in an era obsessed with end-to-end multimodality.
A detailed benchmark pitting vision-capable LLMs (the 'attach PDF and let the model read it' approach) against traditional OCR-based pipelines for long-document question answering has revealed surprising results. Using 30 image-heavy PDFs from the MMLongBench-Doc dataset (171 total questions) and Claude Sonnet 4.5 as the LLM backend, the test compared six configurations. The top performer was LlamaCloud premium with full-context extraction, achieving 59.6% accuracy at $0.1885 per query, closely followed by Azure premium at 58.5% ($0.2051). In stark contrast, the native PDF vision LLM arm—where the model directly processes the PDF as images—came fifth out of six with only 52.0% accuracy and was the most expensive at $0.2552 per query. Even the cheaper OCR arms (Azure basic at 54.4% for $0.1062, Agentic RAG at 53.2% for $0.0827) outperformed the vision approach on both cost and accuracy.
The vision LLM also suffered from a 7% intrinsic failure rate tied to PDF file size, with 12 queries (out of 27 first-pass failures) remaining permanently broken after five exponential backoff retries. These failures were concentrated in two specific PDFs with predictable transport-layer issues. OCR-based arms had a 0% failure rate after retries. Notably, the vision model particularly struggled on chart-heavy and table-heavy pages—precisely the domain where proponents claim 'vision LLMs make OCR obsolete.' However, the benchmark author cautions that the sample (30 docs) is small; only 3 of 15 head-to-head gaps passed McNemar's pairwise test at α=0.05. The vision-versus-OCR finding does survive statistical scrutiny. The full writeup with methodology and data is available at surfsense.com.
- OCR pipelines (e.g., LlamaCloud) achieved 59.6% accuracy vs. 52.0% for a vision LLM on MMLongBench-Doc, with 7% fewer failures due to file‑size limits.
- Cost savings are substantial: $0.19 per query for OCR vs. $0.26 for the vision LLM—a 26% reduction that scales to millions of documents per month.
- Hybrid OCR+LLM architectures remain the most reliable approach for enterprise document QA until multimodal models solve long‑context and truncation issues.
Why It Matters
Document AI’s future is hybrid: traditional OCR pipelines will complement, not be replaced by, vision LLMs for structured PDF understanding.