McGill researchers build AI to decode brain tractography

Researchers from McGill University and collaborators have developed TractoGraphVLM, a unified vision-language framework that tackles four neuroimaging challenges simultaneously: white matter tract bundle classification, text-to-tract retrieval, anatomical captioning, and visual question answering. The framework represents fiber bundles as streamline graphs where nodes encode 3D position and tangent orientation, processed by a General, Powerful, Scalable (GPS) graph transformer architecture.

The model achieves 91.8% bundle classification accuracy and 84.7% retrieval precision (R@1) on held-out test data from the Human Connectome Project. By training on HCP Young Adult subjects and testing zero-shot transfer to HCP Aging datasets, the researchers demonstrated robustness to age-related shifts, with only modest performance drops on discriminative tasks and larger drops on generative tasks. Notably, language supervision enhanced the model's ability to capture anatomical details like hemisphere and fiber family relationships that weren't explicitly labeled during training. The researchers highlight that a single jointly trained model can perform all four tasks effectively while learning transferable neuroanatomy purely from language-aligned representations.