Deep ensemble AI achieves 99.83% malignancy capture in thyroid ultrasound triage
New ConvNeXt-Tiny ensemble with calibration beats dataset shift—but external validation shows limits.
A team of researchers led by Md. Sadibul Hasan Sadib developed a calibrated selective prediction framework using deep ensembles for ROI-based thyroid nodule classification on ultrasound images. The system uses ConvNeXt-Tiny as the backbone with squeeze-and-excitation attention, trained as a five-member deterministic ensemble. Each member undergoes vector-scaling calibration, and the ensemble outputs a mean malignancy probability along with mutual information (MI) as a disagreement score. A three-tier policy based on MI thresholds triages images into No-FNA suggestion, FNA recommendation, or radiologist review, enabling automated decision support while flagging uncertain cases.
On the internal TN5000 dataset (pooled out-of-fold predictions), the ensemble achieved AUC-ROC 0.9395, average precision 0.9715, and near-perfect calibration with ECE 0.0088 and Brier score 0.0813. At 50% nominal MI retention, 7.2% of cases received a No-FNA suggestion—with 98.3% NPV and 99.83% malignancy capture—while 39.9% got an FNA recommendation and 52.9% went to radiologist review. However, on the external TN3K dataset, performance dropped: AUC-ROC fell to 0.7870, ECE rose to 0.1899, and the frozen policy assigned 83.7% to review. Crucially, no malignant image entered the No-FNA pathway, though FNA recommendation PPV declined to 76.6%, underscoring the challenge of dataset shift and the necessity for local recalibration and prospective validation.
- Internal AUC-ROC of 0.9395 and calibration error of just 0.0088 on the TN5000 dataset.
- Under private dataset shift (TN3K), AUC dropped to 0.7870 but 100% of malignant cases were captured via selective referral.
- Three-tier triage policy achieved 99.83% malignancy capture with only 7.2% of cases receiving a 'No-FNA' suggestion.
Why It Matters
Calibrated selective prediction can reduce unnecessary biopsies while catching all malignancies, but local tuning is required before clinical adoption.