DocOCR-Eval: New Framework Selects Best OCR Tool Without Labels

Choosing the right OCR engine for a given document collection is a persistent pain point, especially when labeled data is scarce or expensive to obtain. In their new arXiv paper, Zihan Xu and colleagues from multiple institutions introduce DocOCR-Eval, a correction-based evaluation framework that eliminates the need for ground-truth annotations entirely. The team systematically benchmarked a range of OCR engines and state-of-the-art multimodal large language models (MLLMs) on scanned documents across different domains and languages, revealing wide performance variation.

DocOCR-Eval addresses the selection challenge with a three-stage pipeline: it first uses multiple MLLMs to propose corrections to OCR outputs, then ranks these corrections by consistency, and finally aggregates rankings to produce an overall tool ordering. The key insight is that aggregating corrections from diverse MLLMs progressively improves alignment with rankings derived from ground-truth labels. Extensive experiments show that the framework reliably selects the best OCR tool in realistic, label-limited scenarios, providing a practical blueprint for deploying document parsing systems in production without manual annotation overhead.