Mammography AI fails when mixing datasets, study shows
AI trained on biopsy-heavy mammography datasets actually performs worse than domain-specific data, researchers found.
A cross-dataset study led by Parham Hajishafiezahramini and colleagues at Memorial University Newfoundland exposed a critical flaw in current approaches to training mammography AI models. The team evaluated an EfficientNet-B5 encoder initialized with Mammo-CLIP weights on the Newfoundland and Labrador Breast Screening Dataset (NLBSD), achieving a baseline AUC-ROC of 0.737. However, when they supplemented training data with biopsy-confirmed cases from external datasets like CBIS-DDSM and CMMD, model performance sharply declined across all configurations, dropping to AUC-ROC values between 0.620 and 0.644.
The root cause emerged during diagnostic testing: the model could almost perfectly predict which dataset an image originated from, despite identical preprocessing. This revealed that dataset-specific acquisition protocols, intensity mappings, and construction biases introduced domain shifts that outweighed the benefit of additional positive cases. The findings underscore that pooling heterogeneous mammography datasets naïvely can harm model reliability, calling for domain-aware strategies in medical AI development.
- AI trained on biopsy-heavy external mammography datasets underperformed vs. domain-specific data (AUC-ROC dropped from 0.737 to 0.620–0.644)
- Dataset-specific artifacts persisted even after normalization, causing the model to learn shortcuts based on origin rather than pathology
- Study authors recommend domain-aware strategies for combining mammography datasets to avoid performance degradation
Why It Matters
Medical AI reliability hinges on domain consistency—naïvely mixing datasets can introduce harmful biases that reduce diagnostic accuracy.