In search of truth: Evaluating concordance of AI-based anatomy segmentation models
New method compares models like TotalSegmentator and MOOSE without ground truth data, revealing major performance gaps.
A collaborative research team, including scientists from institutions like Harvard and the German Cancer Research Center, has published a new framework to address a critical challenge in medical AI. With a growing number of AI models designed to segment anatomical structures from CT scans, researchers often lack the 'ground truth' manual annotations needed to evaluate them. Their solution involves harmonizing the outputs of different models into a standard, interoperable format. This allows for consistent, terminology-based labeling of structures, turning a chaotic set of model outputs into a comparable dataset.
To demonstrate the framework's utility, the team applied it to evaluate six prominent open-source segmentation models: TotalSegmentator (versions 1.5 and 2.6), Auto3DSeg, MOOSE, MultiTalent, and CADS. They tested these models on CT scans from the public National Lung Screening Trial (NLST) dataset, focusing on 31 key structures including lungs, vertebrae, ribs, and the heart. The analysis, streamlined through an extension of the popular 3D Slicer platform, revealed significant disparities in model performance. While models showed excellent agreement on segmenting organs like the lungs, several produced invalid or highly inconsistent results for complex structures like individual vertebrae and ribs.
The resources from this study, including harmonization scripts and visualization tools linked from the paper, provide a practical toolkit for the community. This work moves the field beyond simple accuracy metrics, which require perfect manual labels, and towards a concordance-based evaluation. It empowers researchers and clinicians to make informed decisions when selecting a model for a specific task, ultimately improving the reliability of AI-driven analysis in large-scale medical imaging studies and clinical workflows.
- The framework evaluates six AI models (TotalSegmentator, Auto3DSeg, MOOSE, etc.) on 31 anatomical structures from CT scans without needing manual 'ground truth' labels.
- It revealed major performance gaps, with excellent agreement on lungs but poor/invalid segmentations for complex structures like vertebrae and ribs from some models.
- The team released open-source tools, including 3D Slicer extensions and harmonization scripts, to enable standardized model comparison and selection for medical imaging tasks.
Why It Matters
Provides a critical tool for clinicians and researchers to reliably choose the best AI model for analyzing medical scans, improving diagnostic consistency and research validity.