New study reveals two distinct LLM reasoning failure modes via token-level signals

A new paper by Tanvi Thoria, Kiana Jafari, Marc R. Schlichting, and Mykel J. Kochenderfer (submitted to arXiv in June 2026) introduces a framework for diagnosing how language models fail during reasoning. By analyzing token-level uncertainty signals in reasoning traces, the authors identify two empirically distinguishable failure modes. The first, 'committed failure,' occurs when the model quickly locks onto an incorrect reasoning path early in its trace. A key diagnostic is the 'commitment point'—a threshold beyond which considering additional tokens actually degrades failure detection. In contrast, 'persistent uncertainty' characterizes failures where uncertainty accumulates gradually, and the full reasoning trace is needed to best distinguish failing from successful completions.

The framework was validated across 23 model-dataset configurations (including various LLMs and tasks), with its falsifiable predictions holding in 20 of 23 cases—far above chance. The research also demonstrates direct implications for self-consistency techniques: uncertainty signals can complement self-consistency in some failure modes, while in others (e.g., committed failures after the commitment point) self-consistency can be selectively skipped to save compute. This work provides a principled foundation for building more robust detection of LLM reasoning errors, enabling adaptive strategies that know when to trust, when to gather more tokens, and when to apply multiple sampling strategies.