Human annotators still beat LLMs in cognitive engagement coding, study finds
Humans hit kappa up to 0.998; ICL LLMs stuck at 0.609.
Researchers from Tufts and other institutions (Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas) published a CogSci 2026 paper testing how well LLMs can automatically measure cognitive engagement in collaborative discourse. They extended the ICAP framework (Interactive, Constructive, Active, Passive) into a 7-point scale and had trained human annotators label dialogue turns, then compared their results against three LLM-based approaches: zero-shot prompting, in-context learning (ICL), and self-reflective multi-agent systems.
The results show that humans remain the gold standard. Interrater reliability among humans was excellent (kappa = 0.906-0.998), while ICL-based LLMs only achieved moderate agreement (kappa = 0.541-0.609). Refining the framework with human input improved human agreement substantially (delta kappa = 0.10) but gave only marginal gains for ICL LLMs (delta kappa < 0.04). Interestingly, using agent-refined frameworks improved agreement across different LLMs, though it never matched the human-refined framework. The authors suggest that theory-guided human annotation combined with reflective agent feedback is the most promising path forward for scalable, reliable coding of educational discourse.
- Human annotators achieved kappa 0.906-0.998 on the extended 7-point ICAP scale
- In-context learning LLMs lagged with only moderate kappa 0.541-0.609
- Reflective agents improved cross-model agreement but still fell below human-refined frameworks
Why It Matters
For AI-assisted education, this shows human-in-the-loop is still vital for nuanced discourse coding.