MAE with spectral loss beats JEPA for brain MRI disease detection

A team led by Esra Ergün at NYU Langone Health has developed self-supervised foundation models for 3D brain MRI that systematically compare two pretraining paradigms: masked autoencoders (MAE) and joint embedding predictive architectures (JEPA). The key innovation is the introduction of a novel spectral-domain reconstruction loss for MAE and variance-covariance regularization (VCR) for JEPA, designed to capture fine-grained anatomical structures and decorrelated latent features, respectively. The models were trained on heterogeneous single-contrast MRI volumes without modality concatenation, making them contrast-agnostic.

Across five downstream disease detection tasks, the team found that MAE with spectral regularization consistently outperformed JEPA, especially when the discriminative signal involved high-frequency anatomical structures. Conversely, JEPA with VCR was most beneficial when discriminative information spanned multiple decorrelated feature dimensions. This work provides a principled framework for choosing self-supervised objectives based on task structure, promising more accurate MRI-based disease detection without extensive labeled data.