Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores

A team of researchers has developed a novel method to tackle a critical flaw in large language models (LLMs): their tendency to be 'confidently wrong.' The paper, 'Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores,' introduces a technique that scores the agreement patterns between different internal layers of a model during a single forward pass. This provides a compact, per-instance measure of uncertainty without the computational overhead of traditional methods that probe high-dimensional representations.

The new method proves robust across several challenging scenarios. In cross-dataset transfer tests—where a model trained on one task is evaluated on another—it consistently outperformed standard probing techniques, achieving gains of up to +2.86 AUPRC (Area Under the Precision-Recall Curve) and +21.02 Brier score points. It also remained effective under 4-bit weight-only quantization, a common technique for making models smaller and faster, improving over probing by an average of +1.94 AUPRC points. Beyond raw performance, analyzing which specific layer interactions contribute to the score reveals differences in how models like GPT-3 or Llama encode uncertainty internally.

Altogether, this research offers a practical, lightweight tool for developers and researchers. By providing a more reliable signal of an LLM's confidence, it can improve the safety and reliability of AI applications in areas like medical advice, legal analysis, or customer support, where knowing when the model is uncertain is as important as the answer itself.