Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries

Researchers Natalie Perez, Sreyoshi Bhaduri, and Aman Chadha have published a paper introducing the Inductive Conceptual Rating (ICR), a novel metric designed to evaluate the meaning, not just the lexical similarity, of text summaries generated by Large Language Models (LLMs). The work, titled 'Simulating Meaning, Nevermore!', critiques current evaluation methods that rely on statistical approximations like BLEU or ROUGE scores, arguing they fail to capture the relational, context-dependent, and emergent nature of human meaning. The authors propose an interdisciplinary framework integrating semiotics (the study of signs) and hermeneutics (the theory of interpretation) with qualitative research methods to bridge the gap between LLM outputs and human interpretive understanding.

The ICR metric is grounded in inductive content analysis and reflexive thematic analysis, applying systematic qualitative interpretation to assess semantic accuracy and meaning alignment. In an empirical comparison of LLM-generated and human-generated thematic summaries across five datasets, the research found that while models like GPT-4 and Claude achieve high linguistic similarity to reference texts, they consistently underperform on semantic accuracy. Performance improved with larger datasets but remained variable, suggesting LLMs struggle with contextually grounded meanings and the coherence of recurring concepts. This finding challenges the assumption that better lexical metrics equate to better summaries and calls for a paradigm shift in AI evaluation toward frameworks that can assess the conceptual and interpretive depth of machine-generated language.