POPSICLE benchmark suite standardizes cryoET segmentation and localization
Open benchmark datasets tackle the ML bottleneck in electron tomography
Cryo-electron tomography (cryoET) allows direct visualization of macromolecular structures within intact cells, bridging molecular architecture and cellular organization. However, machine learning development for cryoET has been bottlenecked by a lack of standardized, well-annotated benchmarks. Existing evaluations are typically small, task-specific, and isolated, preventing robust cross-method comparisons. To address this, researchers from several institutions (including Schwartz, Ermel, Owens, Zhao, Peck, Hart, Jensen, Carragher, and Kimanius) have introduced POPSICLE, a benchmark suite built from the CryoET Data Portal, an open ML-ready repository of tomographic data, metadata, and annotations.
POPSICLE covers a diverse range of samples: eukaryotic and prokaryotic systems, both purified and fully in situ, plus two task types—dense voxel-wise segmentation and sparse macromolecular localization. As a living data resource, it can expand as new datasets and annotations become available. Baseline experiments conducted by the team show substantial variation in model rankings across different tasks, underlining the need for benchmarks tailored to cryoET's unique characteristics rather than adapting evaluation practices from adjacent biomedical imaging domains. POPSICLE thus provides an open and extensible foundation for reproducible ML evaluation in cryoET, accelerating computational analysis in structural biology.
- Built from the open CryoET Data Portal, ensuring reproducibility and continuous expansion
- Covers both eukaryotic/prokaryotic systems and purified/in situ samples across dense and sparse tasks
- Baseline experiments show model rankings vary dramatically by task, validating the need for cryoET-specific benchmarks
Why It Matters
Standardized benchmarks accelerate ML-driven discoveries in structural biology by enabling fair, reproducible comparisons.