ReportQA uses LLMs to evaluate radiology reports with QAScore

Radiology report evaluation has long relied on natural language generation metrics that lack clinical relevance or clinical efficacy metrics that require heavy manual annotation and cover only limited entities. To overcome these limitations, researchers—including Yiming Shi, Shaoshuai Yang, and colleagues—developed ReportQA, a QA-based evaluation framework that treats radiology reports as context for downstream diagnostic tasks. The team first collected datasets covering multiple imaging modalities and anatomical regions, then built knowledge trees of clinical entities and attributes with guidance from radiologists. Using LLMs, they extract structured information from raw reports and generate QA pairs from predefined templates, applying quality control through self-filtering and report-based filtering. During evaluation, an LLM judge answers the QA pairs using the report as context, producing a QAScore metric that quantifies report quality.

In experiments on multiple state-of-the-art vision-language models, QAScore showed significantly better alignment with radiologist judgments compared to existing metrics like BLEU, ROUGE, or clinical efficacy scores. The study also revealed that current report-based inference paradigms struggle to learn fine-grained clinical representations and exhibit strong negative prior biases, whereas question-driven inference offers a more effective alternative. For reproducibility, the authors released the knowledge trees, structured reports, QA pairs, and pipeline code. This approach promises a more clinically relevant, flexible, and scalable method for automated radiology report evaluation, potentially accelerating the adoption of AI-generated reports in healthcare.