Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

A team of researchers has published a paper titled 'Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection,' proposing a novel method to make large language models (LLMs) reliably assess their own uncertainty. The core innovation is a normalized confidence score derived from output token probabilities—using classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This allows for direct detection of errors and hallucinations without external validation. The framework was tested across seven benchmark tasks and five different LLMs, including Qwen3-4B.

The research makes three major contributions. First, it provides a practical self-evaluation framework that exposes reliable confidence estimates. Second, a theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence, while reinforcement learning methods like PPO, GRPO, and DPO induce overconfidence by exploiting reward signals. Third, the team proposes a corrective technique: post-RL SFT with self-distillation to restore confidence reliability in models trained with reinforcement learning.

The empirical results are significant. Applying SFT improved the average confidence-correctness AUROC (Area Under the Receiver Operating Characteristic curve) from 0.806 to 0.879 on the Qwen3-4B model and reduced calibration error from 0.163 to 0.034. In contrast, GRPO and DPO degraded these metrics. The method's practical value was demonstrated in a retrieval-augmented generation (RAG) system that selectively retrieves external context only when the model's confidence is low. This adaptive approach used just 58% of the retrieval operations to recover 95% of the maximum achievable accuracy gain on the TriviaQA benchmark, showcasing major efficiency improvements.