Linking spatial biology and clinical histology via Haiku

Haiku is a tri-modal contrastive learning model designed to bridge spatial biology, clinical histology, and patient metadata. Trained on 26.7 million patches from multiplexed immunofluorescence (mIF) images, it pairs each patch with matched hematoxylin and eosin (H&E) histology and structured clinical data from 1,606 patients across 11 organ types. By aligning these three modalities into a shared embedding space, Haiku enables cross-modal retrieval — for instance, finding relevant H&E slides from an mIF query or vice versa — with a Recall@50 of 0.611 versus near-zero for baselines. The model also excels at downstream clinical tasks: survival prediction reaches a C-index of 0.737, a 7.91% improvement over unimodal approaches, and zero-shot biomarker inference yields a mean Pearson correlation of 0.718 across 52 biomarkers by fusing retrieval with clinical text descriptions without any task-specific fine-tuning.

Beyond retrieval and prediction, Haiku introduces a counterfactual prediction framework that isolates niche-specific molecular shifts by modifying only clinical metadata while fixing tissue morphology. In a lung adenocarcinoma case study, the model recovered patterns consistent with favorable outcomes: increased CD8 and granzyme B, reduced PD-L1, and decreased Ki67. The authors present these results as exploratory, hypothesis-generating signals, not mechanistic claims. Haiku's tri-modal alignment represents a significant step toward integrating molecular measurements with clinical context for biological exploration, potentially enabling pathologists and researchers to mine large-scale tissue banks for spatial biology insights without needing costly new assays. The paper is available on arXiv (2605.00925).