Claude Sonnet 4.5 vision loses to OCR pipelines in document QA benchmark

A Reddit user benchmarked vision-capable LLMs against OCR-based pipelines on 30 long, image-heavy PDFs from the MMLongBench-Doc dataset, totaling 171 questions. Using Claude Sonnet 4.5 as the underlying LLM, they tested six approaches: four OCR-based (LlamaCloud premium/basic, Azure premium/basic, all with full-context or agentic RAG) and one native PDF vision approach. Premium OCR with full-context led at 59.6% accuracy ($0.1885/query), followed closely by Azure premium at 58.5% ($0.2051). Native PDF vision placed fifth of six with 52.0% accuracy and the highest cost ($0.2552/query).

Crucially, vision underperformed specifically on chart-heavy and table-heavy pages — exactly the areas where proponents argue vision LLMs make OCR obsolete. The native-PDF arm also suffered a 7% intrinsic failure rate (27 first-pass failures, 12 permanently broken) tied to PDF file size, while OCR arms had 0% failure after retries. Statistical testing (McNemar's pairwise) showed only 3 of 15 head-to-head gaps were significant at α=0.05, but the overall vision vs. OCR finding survives. The benchmark is limited by 30 documents but challenges the assumption that vision LLMs can replace OCR for long, complex documents.