Chest X-ray AI models misjudged by flawed evaluation metrics, study finds

A team led by Panagiotis Fytas and 16 co-authors from Cambridge University Hospitals and other institutions systematically investigated how evaluation-reference choices affect model performance and ranking in chest X-ray (CXR) machine learning. They collected paired expert image-derived and report-derived labels for thoracic findings from a clinical cohort at CUH and curated a subset of MIMIC-CXR with expert diagnostic quality ratings. Testing supervised classifiers (ResNet, DenseNet) and vision-language models (MedKLIP, GLoRIA, ConVIRT), they found that changing label sources led to substantial differences in both performance estimates and model rankings. In image quality assessment (IQA), commonly used metrics like SSIM and PSNR often failed to align with expert assessments of diagnostic usability.

The study's results demonstrate that evaluation choices are crucial: they can determine which models and methods appear best and are therefore selected for further development or deployment. The authors stress that evaluation references should be treated as a central component of clinical validity in CXR machine learning, and must be justified with respect to pathology, imaging task, and intended downstream clinical use. This work has significant implications for clinical AI deployment, as models that perform well under one evaluation scheme may be subpar under a clinically more relevant one.