New 'Structural Uncertainty' Framework Reveals LLM Reasoning Consistency
A novel metric goes beyond answer dispersion to evaluate logical reasoning consistency in LLMs.
Large language models (LLMs) often produce the same answer through unstable or contradictory reasoning paths, a failure mode especially common in multi-step deduction. Existing reliability metrics focus on answer dispersion—how much sampled answers differ—but miss a key signal: whether the model consistently ranks its own reasoning candidates. In a paper published at the ICLR 2026 Workshop on Logical Reasoning of LLMs and winner of the best paper award, researchers Baishali Chaudhury, Mengdie Flora Wang, Hyunji Hayley Park, Rahul Ghosh, Sungmin Hong, and Jae Oh Woo introduce structural uncertainty. The framework generates multiple candidate solutions for a query, asks the LLM to judge pairwise preferences among its own outputs, then aggregates those self-preferences into ranking distributions using Bradley-Terry modeling with PageRank. The signal is decomposed into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity.
The framework was evaluated across five LLMs and eight benchmarks spanning logical reasoning, mathematical reasoning, and factual retrieval. The results show that structural signals provide information complementary to traditional answer dispersion: combining the two improves the identification of unreliable instances on logical and math tasks. Interestingly, the two components relate differently to accuracy. Within-trial ambiguity correlates positively with correctness—consistent with settings where multiple plausible solution paths remain competitive—while across-trial instability correlates negatively, signaling unreliable reasoning. On factual retrieval, the structural signal collapsed toward uniformity, diagnosing a regime boundary where reasoning-level consistency evaluation is uninformative. The authors emphasize that structural uncertainty is not a universal confidence estimator, but a regime-sensitive evaluator of logical reasoning consistency. This approach offers a more nuanced understanding of when LLMs are truly consistent versus merely lucky.
- Structural uncertainty uses self-preference rankings and Bradley-Terry with PageRank to measure reasoning consistency.
- Two entropy components: within-trial ambiguity (positive correlation with accuracy) and across-trial instability (negative correlation).
- Tested on 5 LLMs, 8 benchmarks; won best paper at ICLR 2026 Workshop on Logical Reasoning of LLMs.
Why It Matters
Dual-entropy consistency check reveals when LLM reasoning is reliable—critical for high-stakes deployments.