Penn study: Synthetic data underperforms in scarce medical vision domains

A new arXiv paper (arXiv:2608.13729) from University of Pennsylvania researchers examines whether diffusion-based generative models can alleviate data scarcity in specialized vision domains. While synthetic image generation has shown promise on natural image benchmarks like ImageNet, its value in sparse, high-variance real-world domains remained unclear. The team compared two synthetic data extension strategies—distribution modeling and sample perturbation—against non-generative augmentation baselines across five trauma classification tasks using subject-wise train-validation splits. Their finding is sobering: no generative approach consistently outperformed a strong classic baseline, despite producing images that look realistic.

Feature-space analysis identified three recurring failure modes. First, models either memorized training samples or collapsed into narrow output modes. Second, generated images exhibited distributional drift, shifting away from the real clinical data distribution. Third, models produced visually plausible but simplified canonical instances that are easier to classify than actual patient images, making synthetic-data evaluation misleading. These results suggest that for high-stakes domains with expensive data collection, traditional augmentation remains a more reliable strategy, and synthetic data's apparent gains may stem from unrealistic simplification rather than capturing true clinical complexity.