New Office Comprehension Benchmark reveals LLMs score only 59% on real-world Word, Excel, and PowerPoint tasks

A team of researchers (including members from Microsoft) has released the Office Comprehension Benchmark (OCB), the first public benchmark designed to evaluate LLM systems on understanding native Office file formats. OCB addresses a critical gap: while AI models show impressive performance on text-only benchmarks, their ability to parse and reason over actual .docx, .xlsx, and .pptx files—with embedded tables, charts, images, formulas, and app-specific elements like headers, speaker notes, and named ranges—remains largely untested. OCB splits evaluation into two tracks: File Fidelity Q&A, which assesses perception of document structure and visual elements, and Domain Q&A, which requires multi-step analysis and synthesis across real-world industry documents from 12 domains. Each answer is decomposed into atomic, binary-gradable claims scored by an ensemble of LLM judges to ensure reliability.

Results from OCB reveal a significant gap in current AI capabilities. The best frontier system in its default reasoning mode achieves only 59.3% on Domain Q&A, and increasing thinking depth within a product tier yields negligible improvements. Moving to a higher-tier model provides only modest gains, suggesting that raw scale or reasoning depth alone is insufficient for robust Office comprehension. The benchmark exposes specific weaknesses in handling nested tables, cross-document references, and domain-specific jargon. By releasing the dataset, evaluation tools, judge prompts, and a public leaderboard, the authors aim to drive progress in document AI. For professionals who rely on Office suites daily, this benchmark underscores that even the most advanced LLMs have a long way to go before they can reliably automate complex document tasks.