CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark

A new benchmark called CTSCAN, created by researcher Anton Ivchenko, exposes a critical flaw in how medical AI models for chest CT segmentation are typically evaluated. The research reveals that when training and testing data inadvertently mix different CT slices from the same patient—a common practice—performance metrics become severely inflated. In controlled experiments using an FPN with EfficientNet-B0 architecture, models scored 0.6665 in foreground Dice under the flawed "slice-mixed" protocol, but plummeted to just 0.2066 when evaluated with proper "patient-disjoint" splits. This represents a staggering 69% relative performance drop, showing that much of what appears to be AI capability is actually data leakage.

CTSCAN addresses this problem by providing a reproducible research stack that aggregates 89 cases from three public datasets (PleThora, MedSeg SIRM, and LongCIU) with corrected patient-disjoint partitions. The benchmark includes deterministic split manifests, explicit weak-supervision controls, and scripted multi-seed protocol sweeps to ensure evaluations are both rigorous and repeatable. By packaging everything needed for proper validation—from data to figure generation—CTSCAN gives researchers a standardized foundation for developing clinically relevant segmentation models that can generalize to new patients rather than just memorizing training data.