Learning Under Extreme Data Scarcity: Subject-Level Evaluation of Lightweight CNNs for fMRI-Based Prodromal Parkinsons Detection

A new methodological study by researcher Naimur Rahman exposes a critical flaw in how AI models are typically evaluated for medical imaging tasks with limited data. The paper, titled 'Learning Under Extreme Data Scarcity: Subject-Level Evaluation of Lightweight CNNs for fMRI-Based Prodromal Parkinson's Detection,' demonstrates that commonly used evaluation practices in neuroimaging AI research can produce dangerously misleading results. Using fMRI data from just 40 subjects (20 with prodromal Parkinson's, 20 healthy controls), Rahman shows that when researchers use image-level data splits—where slices from the same subject can appear in both training and test sets—models achieve near-perfect accuracy (over 95%) through information leakage rather than genuine learning. This creates a false sense of model capability that wouldn't translate to real clinical use.

When Rahman enforced strict subject-level evaluation—where all data from a given subject stays entirely within either training or testing—performance dropped dramatically to 60-81% accuracy. The study compared several convolutional neural network architectures including VGG19, Inception V3, Inception ResNet V2, and MobileNet V1. Surprisingly, the lightweight MobileNet V1 with significantly fewer parameters demonstrated the most reliable generalization, outperforming deeper architectures in this extreme low-data regime. The research indicates that evaluation methodology and appropriate model capacity matter more than architectural complexity when working with scarce medical data. While limited to a single 40-subject cohort without external validation, this case study provides concrete recommendations for more rigorous AI evaluation in healthcare applications.