PathReportEval benchmark reveals flaws in standard AI pathology metrics

A team of researchers from Stony Brook University and collaborators have released PathReportEval, a comprehensive benchmark designed to standardize evaluation of AI-powered pathology report generation from whole-slide images (WSIs). The field has been fragmented: different studies use different datasets, visual encoders, model architectures, and evaluation protocols, making meaningful comparison nearly impossible. PathReportEval addresses this by providing a modular, plug-and-play framework that standardizes preprocessing, feature extraction, training, decoding, and evaluation across four representative generation methods.

At the heart of the benchmark is the Clinical Report Quality Score (CRQS), a clinically grounded metric that maps generated and reference reports into structured clinical attributes. Unlike traditional natural language generation metrics like BLEU, ROUGE, and METEOR—which primarily reward lexical similarity—CRQS evaluates four complementary dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance. Experiments across three datasets (TCGA, HistAI, and REG 2025) and three pathology foundation encoders (CONCHv1.5, UNI2-h, H-Optimus-1) show that conventional metrics frequently overestimate report quality and fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes. CRQS reveals meaningful differences between models that lexical metrics miss, establishing a reproducible foundation for rigorous evaluation.