Study shows LLM rationales can backfire and reduce user trust

A new study from Xin Sun and colleagues challenges the common practice of having large language models (LLMs) display their step-by-step reasoning to users. Published on arXiv in June 2026, the research tested whether showing rationales helps people calibrate trust or merely persuades them through fluent reasoning. In Study 1, 68 participants evaluated factual verification tasks under varying rationale formats (instant, delayed, on demand), rationale correctness, and certainty framing. The results showed that rationale correctness and certainty framing significantly influenced users' trust in the information, trust in the LLM system, and decision confidence, while presentation format had no reliable effect.

Study 2 used eye-tracking on 54 participants to examine cognitive effort. Incorrect rationales led to longer fixations on supporting evidence and larger pupil diameters, consistent with increased cognitive load. Importantly, showing incorrect rationales actually lowered trust in the LLM system compared to showing no rationale at all—a finding that challenges the assumption that more reasoning is always beneficial. The authors advocate for rationale designs that are selective, evidence-linked, calibrated in certainty expression, and easier to verify. This research has direct implications for how companies like OpenAI, Google, and Anthropic design their AI interfaces.