Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images

A team of 16 researchers led by Aneesh Rangnekar from Memorial Sloan Kettering Cancer Center has tackled a critical challenge in medical AI: transferring pretrained transformer models from CT to MRI modalities. Their paper, published on arXiv, reveals that common assumptions about out-of-domain (OOD) transfer often fail due to two interacting issues. First, zero-padding to match pretrained input dimensions wastes token capacity, diluting attention toward uninformative padding tokens (measured via a new Attention Dilution Index). Second, pretrained features often don't adapt effectively to MRI, even after extensive fine-tuning. Using CT-pretrained hierarchical shifted-window transformers—SMIT and Swin UNETR—the team introduced a tumor-aware augmentation strategy that enriches training data with more varied tumor appearances, plus an anisotropic cropping method that restores token efficiency.

Their results are striking: fine-tuning on a rectal MRI dataset of 247 patients, SMIT achieved a 90.7% detection rate and Swin UNETR 88.7%, significantly improving over baseline transfer methods. This study is among the first to systematically examine when pretrained transformers fail across imaging modalities and offers simple, mechanistically motivated fixes. By addressing both token efficiency and feature adaptation, the work paves the way for more reliable AI-assisted cancer screening and segmentation in clinical settings where MRI is preferred over CT for soft tissue contrast.