SHAP beats LLMs for explaining teaching quality scores

A new study accepted to ACL 2026 systematically compares the faithfulness and transferability of two explanation methods for automated rubric-based scoring: SHAP (Shapley-value attributions) and LLM-generated rationales. The researchers applied their framework to the Quality of Feedback dimension of the CLASS framework using the NCTE corpus of 6,000 annotated transcript segments. Fine-tuned pretrained language models (PLMs) outperformed prompted LLMs in scoring accuracy but showed label compression toward mid-scale scores. In deletion-based tests, SHAP consistently identified sentences that drove model predictions with larger and more coherent prediction shifts than LLM rationales.

Critically, SHAP attributions transferred robustly across different model architectures, while LLM-generated rationales exerted limited and inconsistent influence. The findings suggest that for high-stakes educational assessment, model-agnostic feature attribution methods like SHAP provide more faithful and transferable explanations than prompting LLMs to explain scores. The framework offers a principled basis for evaluating both scoring models and their interpretability in rubric-based language assessment tasks.