Leukemia detection benchmark reveals data leakage inflates AI performance by 4%

A new study by Nisreen Albzour, published on arXiv (2606.24944), exposes a critical flaw in automated leukemia detection research: patient-level data leakage. Many papers report near-perfect results on the C-NMC 2019 dataset, but Albzour shows these are inflated by random image-level partitioning that allows cells from the same patient to appear in both training and test sets. Using strict subject-disjoint folds from 73 patients and an external test set of 1,867 images from 28 unseen subjects, the benchmark compares LightGBM, RBF-SVM, EfficientNet-B0, EfficientNet-B1, and ViT-Tiny. A random-versus-subject-disjoint ablation confirms that random splitting inflates AUROC by about 0.04, even in conservative frozen-feature settings.

Under honest evaluation, EfficientNet-B1 emerges as the top performer with AUROC 0.913, sensitivity 0.87, specificity 0.80, and a calibrated expected calibration error (ECE) of just 0.024 after temperature scaling. In contrast, frozen-feature classifiers and ViT-Tiny showed high sensitivity but poor specificity, indicating a tendency to over-predict malignancy. The study also assesses model calibration—a crucial aspect for clinical deployment—using ECE and Brier scores. These findings caution against image-level evaluation on C-NMC 2019 and provide a reproducible, calibration-aware benchmark to guide future work in reliable AI-assisted leukemia diagnosis.