Granuscore: A reference-free measure to quantify text granularity in QA

A team of researchers (Ellinger, Fichtl, Anschütz, Groh) from the Technical University of Munich has released Granuscore, a novel reference-free metric that measures the granularity of natural language—from fine-grained specifics to broad abstractions—without requiring any human-annotated gold standards. Published on arXiv (2605.26620), Granuscore leverages structural properties of hierarchical embedding spaces to assign a single score that captures how detailed or coarse a piece of text is.

The metric reliably recovers known hierarchical orderings on the Granola-EQ dataset and detects expected granularity shifts across different discourse contexts. Notably, Granuscore explains non-linear variation in sentence specificity beyond what sentence length alone can capture, making it more robust than word-count proxies. When applied to four popular question-answering benchmarks, Granuscore revealed systematic differences in granularity between questions, gold answers, and model outputs, offering a principled lens to characterize dataset difficulty and model behavior. This positions Granuscore as a broadly applicable, scalable tool for researchers analyzing text granularity in domains from summarization to QA evaluation.