Rad-JEPA 3D: Compact 4B-parameter model sets SOTA in 3D CT analysis

Radiology AI faces a fundamental challenge: unlabeled CT volumes are abundant, but expert annotations are scarce. Existing volumetric encoders often lose the coarse spatial structure needed for downstream tasks like organ disentanglement and abnormality detection. Rad-JEPA 3D tackles this with a joint-embedding predictive approach — it learns representations by predicting latent features of a complete scan from a masked view. At its core lies a hybrid H-Mamba encoder that fuses a Mamba state-space branch (modeling inter-slice continuity) with a grouped-query attention branch (capturing cross-plane context), combined via a lightweight per-token router. To further enrich representations, the authors propose Hidden States Orthogonal Regularization (HSOR), which aligns student-teacher hidden states and reduces feature redundancy across encoder layers.

Pretrained on roughly 120,000 CT scans, Rad-JEPA 3D achieves state-of-the-art results despite its compact 4.0B parameter size. It matches larger models on closed-ended visual question answering and obtains the best average spatial-reasoning score on the Spatial-Med benchmark. Ablation studies confirm that both the hybrid block and HSOR contribute complementary gains, and that the induced spatial structure can substitute for raw language-model scale on volumetric reasoning tasks. This work opens the door for more efficient and accurate AI-assisted radiology workflows, especially in settings where compute resources are limited.