Claude Sonnet 4.5 vision falls short vs OCR pipelines on document QA benchmark
A recent benchmark reveals that a tailored OCR pipeline not only beats a leading vision LLM on document question answering — it does so at a lower cost per query, challenging the assumption that general-purpose AI will soon replace specialized tools.
A new benchmark challenges the claim that vision-capable LLMs make OCR obsolete for document question answering. Testing on 30 image-heavy PDFs from MMLongBench-Doc (171 questions), the author compared proprietary vision LLMs (Claude Sonnet 4.5 reading PDFs natively) against OCR pipelines with layout extraction. The top performer was LlamaCloud premium + full-context at 59.6% accuracy ($0.1885/query), followed closely by Azure premium + full-context at 58.5% ($0.2051/query). The vision-native approach managed only 52.0% accuracy while being the most expensive at $0.2552/query. Vision particularly struggled on pages with charts, tables, and dense layouts—exactly the use case where proponents claimed it would excel.
Beyond accuracy, the vision pipeline suffered a 7% intrinsic failure rate on large PDFs that persisted after exponential backoff retries (12 permanently failed out of 27 initial failures). OCR-based arms had 0% failure after retries. Statistical analysis (McNemar's test at α=0.05) showed that while most head-to-head gaps were within noise, the vision-versus-OCR difference was significant. The author notes caveats: the 30-document sample is small, and the benchmark covered only one vision model. However, for professionals handling document-heavy workflows, the results indicate OCR with layout extraction remains more reliable, accurate, and cost-effective than current vision LLMs.
- OCR pipelines achieved 59.6% accuracy at $0.1885/query, outperforming a vision LLM's 52.0% accuracy at $0.2552/query — a ~15% relative difference in accuracy and ~26% lower cost.
- Vision LLMs have a 7% intrinsic failure rate on large PDFs likely due to context window limits; specialized OCR services like Google Document AI and AWS Textract avoid this by processing pages individually.
- The document AI market is projected to exceed $4 billion by 2027, indicating strong enterprise demand for specialized extraction tools that general vision LLMs currently cannot match for structured documents.
Why It Matters
The benchmark exposes the accuracy-cost gap between specialized OCR and general vision LLMs, shaping enterprise adoption decisions in the $4B document AI market.