New Exam-Style Test Shows AI Gets Distracted by Cluttered Documents

AI that can look at images and read text at the same time (called vision-language models) is now being used to scan receipts, reports, diagrams and scanned paperwork. But most existing tests for this skill were too easy in a specific way: they either used one clean, single page with no distractions, or they tested tough visual puzzles on almost no text. Nobody had properly tested the messy reality most of us live in — a 40-page document with charts, photos and plenty of pages that don't matter.

So two researchers, Yongqi Yu and Yu Zhang, built TestHallVQA. It uses real scientific exam material, spread across multiple pages and images, and asks the AI to answer the kinds of questions a student would face. Crucially, they can deliberately add layers of redundant, irrelevant content, then watch what happens. They also designed a new scoring method, F1-R², that grades two things at once: whether the AI got the answer right, and whether it found the right evidence instead of guessing from the wrong page.

The results were unflattering. Adding unnecessary visual clutter visibly dragged down performance, and many models failed to point at the correct supporting passage even when they stumbled onto a correct answer. In plain terms: the AI looks smart on a tidy one-page task and then loses the plot when the document gets long and noisy — which is exactly what real-world documents look like.

The catch: this is still a lab test using exam questions, not a scorecard for the specific AI assistant you use at work, and a bad score doesn't mean AI is useless at reading documents — just that it needs checking. But the direction is clear. Companies are racing to sell AI that reads your paperwork, and this research shows that honesty about where these tools break down is still catching up.