New AI model improves medical VQA with DINO-based attention

Researchers from MIT, Harvard, and other institutions have developed an attention-guided encoder-decoder for longitudinal medical visual question answering (VQA) using chest X-rays. The model, presented at CVPR 2026 Workshop PHAROS-AIF-MIH, addresses the challenge of reasoning about anatomical differences between current and reference time point images. Instead of conventional direct contrast, the team adds a lightweight affine registration module to co-register the current image to the reference, with a small regularizer to reduce nuisance motion. The registered pair then feeds into a frozen DINO-based mask generator and a trainable adaptive mask generator, producing masks applied to original image pairs. Masked images are re-encoded and concatenated with text features for a multimodal transformer decoder that outputs answers.

To stabilize learning and clarify change signals, the framework incorporates auxiliary objectives inspired by DINO-v3: a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss that enhances representation geometry. On the Medical-Diff-VQA benchmark, the model delivers strong BLEU, ROUGE-L, CIDEr, and METEOR scores while offering intrinsic interpretability through shared saliency masks. This supports saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning. The approach also demonstrates a paradigm for optimizing both supervised and unsupervised learning objectives simultaneously, showcasing how vision foundation models can be effectively utilized in biomedicine.