Foundation models show strong out-of-distribution performance for multimodal cancer diagnosis

A new study from Harvard, Stanford, and Cambridge systematically evaluated foundation model (FM) representations for multimodal cancer analysis across two real-world commercial cohorts: IH-BC (breast cancer) and IH-NSCLC (non-small cell lung cancer). The researchers benchmarked five different FMs (including pathology-specific vision models and genomic language models) on eight downstream classification tasks using whole-slide images and transcriptomic profiles. Key findings showed that image and omics representations carry complementary predictive signals, and that FM features generalize well even under distribution shift—outperforming task-specific models in several cases.

Multimodal fusion strategies (late, intermediate, and hybrid) were tested across paired representations, revealing that fusion only improves over the best single modality when no single signal dominates—a nuanced insight for clinical deployment. Trustworthiness was assessed via conformal prediction, which produces prediction sets with guaranteed coverage. Critically, in most cases where the point prediction was wrong, the true diagnosis remained recoverable within the prediction set, underscoring the value of uncertainty-aware inference for clinical decision support. The work provides a reproducible framework for evaluating foundation models in computational pathology.