MedReCo-VLM boosts radiology follow-up accuracy by up to 46.5%

Medical imaging AI has excelled at analyzing isolated scans but fails to mimic how radiologists work—comparing current images with prior studies or similar reference cases. To bridge this gap, Tengfei Zhang and colleagues from Shanghai Jiao Tong University propose MedReCo, a vision-language framework built for entity-aware cross-image reasoning. They constructed MedReCo-DB, a large-scale comparative imaging resource derived from routine clinical reports, containing over 690,000 images from more than 160,000 patients across eight institutions, four countries, and seven imaging modalities. Reports are decomposed into anatomical structures, abnormal findings, and pathological conditions to supervise retrieval and comparative question answering. The framework comprises MedReCo, an entity-aware encoder for controllable retrieval of clinically analogous cases, and MedReCo-VLM, a vision–language model for generative interpretation of interval change.

In extensive evaluations spanning internal, external, and cross-center settings, MedReCo achieved the highest Recall@1 in all 12 internal retrieval settings and improved external retrieval by a mean of 6.0 percentage points. For clinically confusable differential groups, it consistently outperformed strong baselines. MedReCo-VLM led all comparative generation benchmarks, improving longitudinal follow-up accuracy by 14.5–46.5 percentage points on chest radiographs and 13.0–27.9 percentage points on CT. These results demonstrate that entity-aware comparative reasoning can be learned from routine clinical data at scale, offering a more clinically aligned foundation for medical imaging AI.