Financial AI models overconfident on charts; trained probes fix trust

A new arXiv study (2608.06532) tackles a critical problem in financial AI: large vision-language models (LVLMs) that read charts and documents often produce authoritative-sounding answers without actually grounding them in the visual data. Researchers Khanmohammadi, Kaur, Smiley, Brugere, and Ghassemi tested whether confidence scores from these models can be trusted enough for automated decision-making or deferral to human reviewers. They evaluated seven confidence estimators — three inference-only baselines and four trained internal probes — across five open-weight LVLMs on three financial VQA benchmarks (including one bilingual). Critically, every probe was trained only on natural images and applied to finance with no adaptation, making the results a strict test of out-of-distribution transfer.

The findings are stark: inference-only baselines rank correct answers above incorrect ones reasonably well, but they are severely overconfident, with calibration errors far beyond any usable threshold. Only trained probes produce scores you can actually set a threshold on. Reliability is also highly structured, not global — the best estimator shifts by model and task, with no estimator leading more than 8 of 20 model-condition cells. A controlled bilingual contrast revealed that apparent language robustness is largely a composition artifact that disappears when models are analyzed individually. When cast as deferral under an error budget, the authors found that how much can be safely automated is set primarily by the model's raw competence, and confidence scoring only narrows that. At a strict 5% error budget, deferral clears a sizable share of easy examples and almost none of the hardest ones. Notably, a grounding-aware probe uniquely lowers its confidence when the model gives answers without using the figure, making it possible to detect non-grounding rather than mistake it for a fluent guess.