ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts

Researcher Rongtian Ye has introduced ChartDiff, the first large-scale benchmark designed to test AI models on the complex task of comparative reasoning across multiple charts. The dataset consists of 8,541 chart pairs spanning diverse data sources, chart types, and visual styles, each annotated with summaries describing differences in trends, fluctuations, and anomalies. These summaries were generated by large language models (LLMs) and then rigorously verified by humans, creating a robust ground truth for evaluation.

Using ChartDiff, the research team evaluated three categories of models: general-purpose vision-language models, chart-specialized models, and pipeline-based methods. The results revealed a significant finding: while specialized and pipeline-based methods achieved higher scores on traditional lexical overlap metrics like ROUGE, they scored lower on human-aligned evaluations. In contrast, frontier general-purpose models like GPT-4V achieved the highest quality when assessed by GPT-based metrics, highlighting a clear mismatch between automated scores and actual summary usefulness.

The benchmark also identified specific pain points for current AI. Multi-series charts proved to be a major challenge across all model families, indicating that complexity in data visualization remains a significant hurdle. However, the study found that strong end-to-end models were relatively robust to differences in underlying plotting libraries (like Matplotlib vs. Plotly). Overall, ChartDiff establishes that comparative chart reasoning is a distinct and difficult task that current models have not mastered, providing a crucial new tool for driving research in multi-document visual understanding.