New NLI-hypergraph audit catches bad LLM reasoning LLMs miss
LLMs as judges over-accept fluent but weak reasoning; new framework finds flaws.
A new paper from Guneet Singh Kohli and colleagues introduces a reference-free framework for auditing the reasoning in LLM-generated answers, addressing a critical gap in high-stakes domains like medicine. The method decomposes a model's reasoning trace into segments, labels local premise-target relationships using Natural Language Inference (NLI), and builds a hypergraph structure. It then performs a deterministic backward AND-OR search to assign audit labels to each segment, revealing how every piece of reasoning is grounded within the response—without needing a ground-truth answer.
The framework was tested on two benchmarks: Hard2Verify for deductive mathematical reasoning and UroReason, a new physician-annotated dataset of LLM reasoning traces from real clinical cases. Results show that the NLI-hypergraph audit provides a more reliable evaluation signal than direct LLM-as-judge baselines. In the medical setting, state-of-the-art LLM judges frequently over-accepted fluent but weakly grounded reasoning segments, making the framework especially valuable for clinical applications. The authors plan to release UroReason via API and open-source their code.
- Decomposes LLM reasoning traces into segments and labels premise-target relations using Natural Language Inference (NLI).
- Organizes relations into a hypergraph and runs backward AND-OR search for segment-level audit labels.
- Outperforms LLM-as-judge baselines, especially in medical reasoning where LLMs over-accept flawed segments.
Why It Matters
A more trustworthy way to verify LLM reasoning in high-stakes fields without needing reference answers.