Claude Sonnet 4.5 vision falls short vs OCR pipelines on document QA benchmark

A controlled test pitted a state-of-the-art vision model against a purpose-built OCR pipeline on 30 image-heavy PDFs with 171 questions. The OCR system achieved 59.6% accuracy at $0.1885 per query, while the vision model reached only 52.0% accuracy at a higher $0.2552 per query. The vision model particularly struggled with charts and tables, and exhibited a 7% intrinsic failure rate on large PDFs — likely due to context window limitations. These results confirm that despite rapid progress in multimodal LLMs, extracting fine-grained text from complex layouts remains a domain where decades of OCR optimization still hold the edge.

The landscape of document AI is dominated by three major cloud services: Google Document AI, Microsoft Azure AI Document Intelligence, and AWS Textract. Each offers prebuilt and custom models that combine OCR with structured extraction, achieving high accuracy on invoices, forms, and tables. These specialized services are designed for the very task the vision LLM tried to tackle — but they do it by breaking the problem into focused steps: image preprocessing, text detection, layout analysis, and key-value extraction. The document AI market, projected to exceed $4 billion by 2027 with a 25% CAGR, shows that enterprises are already investing heavily in these dedicated pipelines. The benchmark underscores why: for many real-world use cases, specialized tools deliver both better accuracy and lower cost than a jack-of-all-trades vision model.

The obvious narrative — that general vision LLMs will soon make OCR obsolete — misses a deeper point. The two approaches serve fundamentally different needs. OCR pipelines excel at pixel-perfect text extraction from structured documents, where every digit or table cell matters. Vision LLMs, on the other hand, shine at holistic understanding: reasoning about the content of an image, answering questions that require synthesis, or handling unstructured visuals like photographs. The real insight from this benchmark is that the cost-accuracy tradeoff favors specialization for high-volume, predictable document tasks. But the hidden risks also matter: the test used only 30 PDFs and 171 questions, and the vision model may have been a typo for an earlier version (Claude 3.5 Sonnet) whose 200K token context could have been exceeded. Future models with larger contexts or hybrid architectures that combine OCR preprocessing with LLM reasoning could narrow the gap dramatically.

The bottom line is clear: enterprises evaluating document AI should not default to the latest vision LLM. For structured document QA, specialized OCR pipelines remain the most cost-effective and accurate choice today. Yet, the trend toward larger contexts and improved fine-grained extraction means the gap will likely shrink. The smartest strategy is to watch for hybrid models that merge OCR's precision with an LLM's reasoning — and to bet on vertical solutions that solve a specific problem well, rather than on general-purpose AI alone.