AI Medical Scans May Cheat by Using Hidden Clues
New study reveals AI can cheat on medical image tests, raising doubts about real-world accuracy.
systems that help doctors analyze medical images, like colonoscopies and breast ultrasounds, are often tested with captions describing the lesion. These captions might say 'a large polyp in the upper left' – but the size and location were taken from the very mask the AI is supposed to predict. So the AI learns to use these hints instead of the image itself. This is called 'extent leakage.'
In tests, when researchers kept only the size word or location phrase, the AI recovered most of its performance boost. Removing these hints caused a big drop. The AI even segmented a polyp called a rock just as well, showing it ignores other words. On breast ultrasound, nearly half the captions also named the diagnosis, giving another unfair advantage.
This matters because AI that cheats on tests may fail in real hospitals. When the researchers retrained models without these hints, the models' ability to segment from the image alone weakened. The leak can be measured without training a model, since caption words alone predict lesion area. Without the test mask, the best method reached 74.59 Dice on colonoscopy versus the published 77.23.
The study proposes a new evaluation protocol that separates what a caption says about the image from what it copies from the label. This ensures AI is tested fairly and performs well in real-world settings.
- AI models use hidden clues from captions to boost test scores, not just the image.
- Removing these clues caused performance to drop significantly, showing reliance on them.
- New testing methods are needed to ensure AI works without cheating.
Why It Matters
If AI cheats on tests, it may miss real lesions, leading to misdiagnoses and harming patient care.