How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
Research on 712 teacher interviews shows AI judges miss nuance, offering a practical guide for researchers.
A new study from researchers Songhee Han, Jueun Shin, and colleagues tackles a critical question in AI-assisted research: can we trust LLMs to judge the quality of interpretive analysis? The team tested five leading inference models—Cohere's Command R+, Google's Gemini 2.5 Pro, OpenAI's GPT-5.1, Meta's Llama 4 Scout-17B Instruct, and Alibaba's Qwen 3-32B Dense—on 712 conversational excerpts from K-12 math teacher interviews. Using AWS Bedrock's LLM-as-judge framework, they generated automated ratings across five metrics and compared them to evaluations from trained human raters.
The results reveal a significant gap between AI and human judgment. While LLM-as-judge scores captured broad directional trends at the model level, they showed substantial divergence in score magnitude, especially for non-literal and nuanced interpretations. Among the automated metrics, 'Coherence' aligned best with human ratings, but 'Faithfulness' and 'Correctness' were systematically misaligned at the individual excerpt level. The study concludes that LLM-as-judge methods are currently better suited for screening out underperforming models in a workflow than for replacing human evaluators, offering a practical, evidence-based guide for qualitative researchers looking to integrate AI tools systematically.
- Tested 5 major LLMs (GPT-5.1, Gemini 2.5 Pro, etc.) on 712 teacher interview excerpts using AWS Bedrock's judge framework.
- Found LLM judge scores diverge significantly from human ratings on nuanced interpretations, with Faithfulness and Correctness metrics most misaligned.
- Provides a practical framework: Use LLM-as-judge for initial model screening, not as a replacement for human qualitative judgment.
Why It Matters
Gives researchers a data-backed method to select AI models for analysis, preventing over-reliance on flawed automated scores.