New Study Finds AI's Confidence Is Unreliable When Images Get Blurry

Many AI tools now look at photos and answer questions about them — read a medicine label, spot a crack in a pipe, describe a scene for someone who can't see it. These systems also report how sure they are. That number matters, because we use it to decide when to trust the AI and when to check with a human. This paper asks a simple but important question: when the picture gets worse, does the AI's confidence behave sensibly?

The answer was largely no. Using a small open model (Qwen2.5-VL-3B), the researchers built 176 picture-question tests and then hid the specific regions the question depended on, one step at a time. The AI's confidence broke its expected pattern 43.6% of the time, and 92% of the examples contained at least one moment where confidence moved the wrong way. As a control, they hid equally large but irrelevant areas: accuracy fell 28.2 percentage points for important regions versus just 0.6 points for unimportant ones, confirming the AI really was relying on what got hidden.

Their proposed fix was modest. They trained a small extra layer that watches the model's internal state and penalizes confidence that moves out of order. Violations dropped from 0.330 to 0.303 when regions were masked, and from 0.449 to 0.402 under blur. But on standard measures of ranking which answers to trust, the new layer was no better than the original confidence — and the original was still stronger. The authors say so plainly.

So this is careful, early-stage work, not a product. It used one small model, 176 examples, and lab conditions. The bigger point is the direction: as AI moves into cars, hospitals, and photo apps, an AI that says 'I'm confident' while guessing is a real safety problem. Teaching machines to know when they don't know may matter more than making them smarter.