New framework makes MLLMs reliable chart data extractors

A new research paper accepted at CHI'26 tackles a persistent problem in multimodal AI: extracting accurate numerical data from chart images. While multimodal large language models (MLLMs) can generally interpret charts, their ability to recover exact values—especially from charts without visible data labels—has been unreliable. The authors built a benchmark featuring diverse, real-world charts without labels to systematically evaluate this capability. Their results reveal a clear pattern: current MLLMs are good at reconstructing the structure of data tables (e.g., row and column headers) but struggle with precise value recovery, often hallucinating or misreading numbers.

To address this, the researchers propose a human-inspired progressive learning training framework that teaches MLLMs to extract chart data in stages, mirroring how people read charts. Their approach significantly improves numerical accuracy, achieving state-of-the-art performance with a 7B-parameter model. A user study further demonstrates that the model can effectively power mixed-initiative workflows—where humans and AI collaborate—making chart data extraction both reliable and efficient. This work offers a practical path to automating a tedious task in data science, reproducibility, and visualization redesign.