Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents

Vision-Language Models (VLMs) often misread values, hallucinate details, and confuse overlapping elements in charts because they rely solely on pixel interpretation—treating interactive charts as static images. This creates a 'Pixel-Only Bottleneck' where agents lose access to the structured specification that encodes exact values. To solve this, researchers from William & Mary and Oak Ridge National Laboratory introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines spec-grounded introspection (querying the underlying specification for deterministic evidence) with view-grounded interaction (manipulating the chart view to resolve visual ambiguity).

To evaluate IVG without VLM bias, the team created iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection alone improves data reconstruction fidelity, while combining introspection with interaction achieves the highest QA accuracy (0.81), with a +6.7% gain on overlapping geometries. The researchers further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time, suggesting a new path for reliable AI-powered data analysis.