The System Hallucination Scale (SHS): A Minimal yet Effective Human-Centered Instrument for Evaluating Hallucination-Related Behavior in Large Language Models

A team of researchers from the Medical University of Graz and partners has published a novel framework for evaluating a critical flaw in large language models: hallucinations. Their tool, the System Hallucination Scale (SHS), is a lightweight, human-centered survey instrument designed to capture how hallucination-related behaviors—like factual unreliability, incoherence, and misleading presentation—actually manifest from a user's perspective during realistic interactions. Unlike automated detectors or benchmark metrics, the SHS provides a rapid, interpretable, and domain-agnostic way for people to score an AI's output, inspired by established tools like the System Usability Scale (SUS).

In a real-world evaluation involving 210 participants, the SHS demonstrated high practical utility. Statistical analysis confirmed its clarity, coherent response behavior, and strong construct validity, with an internal consistency score (Cronbach's alpha) of 0.87 and significant correlations between its measured dimensions. The researchers position the SHS not as a replacement for automated metrics, but as a complementary tool for comparative analysis, iterative system development, and ongoing deployment monitoring. It allows teams to quickly gauge user-perceived reliability during the development cycle or after a model is launched.