PulseBench-Tab: New multilingual benchmark tests table extraction with graph-based scoring

PulseBench-Tab is a new open benchmark designed to evaluate table extraction from document images in a multilingual, real-world setting. Created by researchers, it comprises 1,820 human-annotated tables spanning 9 languages (including English, Chinese, Arabic, and Russian) across 4 scripts: Latin, CJK (Chinese-Japanese-Korean), Arabic, and Cyrillic. The tables come from 380 real source documents such as financial filings, government reports, and regulatory disclosures, offering a diverse and challenging test set. Table sizes vary widely from 2 to 1,183 cells, with nearly half (48.1%) containing merged or spanning cells—a common yet difficult scenario for extraction systems.

Alongside the dataset, the authors propose T-LAG (Table Logical Adjacency Graph), a novel evaluation metric that rethinks how table extraction accuracy is measured. T-LAG represents each table as a directed graph where nodes are cells and edges capture logical adjacency. It then computes a single fidelity score by solving an optimal bipartite matching between the extracted and ground-truth graphs, combining structural and content correctness. This approach addresses limitations of traditional metrics like TEDS or cell-level accuracy. The authors evaluated 9 commercial and open-source table extraction systems (including Amazon Textract, Microsoft Azure, and Tesseract) and provide per-language performance breakdowns. The entire dataset, scoring code, and all provider outputs are publicly available to foster reproducibility and fair comparison.