ChartWalker benchmark exposes major gaps in cross-chart AI reasoning
New benchmark reveals current RAG systems struggle with multi-chart analysis
A team of researchers from multiple institutions has released ChartWalker, a novel framework designed to benchmark cross-chart retrieval-augmented generation (RAG) tasks. Unlike existing benchmarks that rely on structured tables or simple key-point extraction—which often leads to lexical overlap and inconsistent reasoning—ChartWalker builds hierarchical knowledge graphs that preserve the analytical structure of charts. It then uses structure-aware sampling to synthesize semantically coherent, multi-hop reasoning paths with explicit control over query difficulty and granularity. The resulting benchmark, ChartWalker-Bench, covers diverse domains and cross-chart query types.
When evaluated across major RAG paradigms, ChartWalker-Bench revealed substantial performance gaps, underscoring its difficulty and utility. The team also released ChartWalker-Agent, an agentic baseline to facilitate further analysis and inspire future system design. This work highlights a critical blind spot: current AI systems struggle to reason across multiple charts, a task essential for complex analytical work in science, business, and policy. The paper is available on arXiv (2606.23997) and includes code and data for community use.
- ChartWalker uses hierarchical knowledge graphs to preserve analytical structure of charts
- ChartWalker-Bench spans diverse domains with synthetically generated multi-hop reasoning paths
- Evaluations across major RAG paradigms show significant performance gaps, highlighting benchmark difficulty
Why It Matters
Exposes how poorly current AI handles cross-chart reasoning, critical for data-driven decision-making