LLMs score 96% on CBT exams but fail to apply therapy effectively

A new study accepted at ACII 2026 exposes a critical gap in LLM performance: while models like GPT-4o and Llama 3 achieve up to 96% accuracy on Cognitive Behavioral Therapy (CBT) licensing exam questions, they cannot effectively apply CBT in real patient interactions. The authors, led by Vaishnavi Sinha, analyzed 14 case studies from the RealCBT dataset across three open-weight LLMs, finding that models default to "validation and reflection" responses regardless of user needs, ignoring structured CBT techniques like Socratic questioning or alternative perspectives.

To quantify this failure, the team developed a knowledge-guided framework that decomposes user narratives into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts. They introduced the Protocol Leverage Force (F), a behavior-level metric that measures how far an intervention shifts a model from its default response. Even with Multiple Chain-of-Thought (MCoT) prompting, the behavioral change was only about 1.2-1.3%, confirming that CBT knowledge alone does not translate to effective application. The paper provides the affective-computing community with instrumentation to understand where LLMs fall short in controlled therapeutic reasoning.