arXiv study: 5-fold cross-validation beats holdout for medical image HPO

Hyperparameter optimization (HPO) can make or break deep learning image classifiers, but the choice of validation signal — how you split your precious training data — has received little empirical scrutiny. Ljubomir Buturovic addresses this gap in the preprint arXiv:2608.14705, comparing three HPO protocols: fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). He measures absolute performance-estimation error (AEE), the gap between the winning configuration's validation AUROC and its real test AUROC, while holding search space, sampler, training procedure, architecture, and test set identical across all protocols.

Buturovic tested on three public datasets covering both medical and natural imaging: RSNA pneumonia chest radiographs, binarized HAM10000 skin lesions, and 200-class Tiny ImageNet, using ResNet-18 on all datasets and ViT-S/16 on RSNA. On the medical datasets, every point estimate favored 5-fold CV over both holdout approaches, with AEE reductions largest at small sample sizes (ranging from 100 to 3,000 development samples) and shrinking as n grew. The pattern held even under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Interestingly, fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, though that finding was less robust. The takeaway: for small-sample medical image classification, use cross-validation-based HPO when computational resources permit — the extra compute buys a substantially more reliable estimate of how the model will actually perform in the wild.