Study reveals flaws in medical AI transfer learning metrics
Medical AI transferability rankings flip with tiny dataset changes, study finds
Transferability estimation (TE) metrics are meant to predict which pre-trained model will perform best on a medical imaging task—but a new study shows their rankings are fragile. Small changes to the target dataset, such as different sample sizes or random seeds, shifted the rankings. The choice of evaluation metric used as the reference also affected results, making it harder to fairly assess TE metrics. Overall, agreement between TE metric rankings and reference rankings was low. The researchers open-sourced their code, model checkpoints, and data splits.
- TE metrics for medical imaging are highly unstable—small dataset changes flip model rankings unpredictably
- Team tested 1M+ configurations across ImageNet and medical datasets, finding low agreement with real performance
- Code, model checkpoints, and data splits are open-sourced at provided URL
Why It Matters
Undermines confidence in AI deployment in healthcare, where model selection directly impacts patient outcomes and diagnostic accuracy.