ChartProbe proves VLMs can learn complex chart reasoning from simple perception skills alone

Vision-language models (VLMs) like GPT-4V and LLaVA still stumble on charts that demand visual reasoning over plotted values. The usual fix is to pile on more complex reasoning supervision, but a new study from Georgetown researchers Mahsa Khoshnoodi and Sarah Adel Bargal flips that assumption. Their ChartProbe framework generates diagnostic questions directly from the code that renders each chart, guaranteeing exact gold answers with zero human annotation. Instead of blaming a reasoning deficit, ChartProbe isolates three underlying skills: perception (reading the plotted marks), grounding (linking marks to labels), and simple reasoning (ranking, totals, differences). Each failure is attributed to exactly one skill.

ChartProbe enables an intervention no prior work attempted: withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, then measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, training on perception and grounding alone recovered much of complex reasoning on questions the models never saw in training—at no reasoning-data cost. The gains held out-of-distribution across an unseen chart type (pie charts), the human-written ChartQA benchmark, and a non-chart visual domain (CLEVR). This suggests that complex visual reasoning can be improved not by adding more complex supervision, but by teaching models to read images more reliably in the first place.