ICML 2026 study reveals LLM high-confidence errors are often stable, not fragile

A new paper by Akira Okutomi, accepted at the EIML@ICML 2026 workshop, investigates why large language models sometimes produce high-confidence errors that don't behave like typical 'fragile' mistakes. The author introduces the concept of 'stable miscalibration' — confident wrong answers that remain locally stable under small perturbations. This directly challenges the prevailing view that overconfident errors are always the result of unstable internal inference or random noise. The study combines two diagnostics: a label-aware output-level audit score that ranks domains by confidence variation and overconfident mistakes under a forced-answer baseline, and an internal sensitivity probe that measures hidden-state movement during generation.

On a multi-domain binary factual audit set, the audit score successfully tracked where abstention-aware self-critique reduces decision loss, though direct labeled baselines ranked the same gain more strongly. Internally, the researcher tested self-critical prompting on three open-weight models and found that it consistently reduced hidden-state sensitivity across layers. This suggests that prompting induces local stabilization, rather than just output-level abstention patterns. Crucially, the results do not imply true calibration: audit-defined overconfident errors were not clearly more locally sensitive than confidently correct answers. This means some high-confidence errors may be stable and systematically miscalibrated — a more dangerous failure mode than simple fragility, because they resist both perturbation-based detection and self-critique.