New ECE framework lets AI abstain from weak fact-checks

A new paper from Dekun Yang tackles a hidden flaw in LLM-based fact-checking: systems confidently declare 'true' or 'false' even when supporting evidence is weak, sparse, or internally inconsistent. The proposed framework, Evidence Chain Evaluation (ECE), lets the model abstain by returning an 'uncertain' verdict when evidence quality doesn't justify a binary decision. ECE operates as a tool-using verification agent that gathers evidence through web search, scholarly search, and executable checks, returning structured verdicts with confidence scores and source-level metadata.

On the custom ECE-Bench benchmark, ECE achieves 91.6% standard accuracy, 93.7% coverage, and 97.8% selective accuracy on the claims it does answer. Crucially, it defers only 6 out of 95 cases – five of which fall under the lowest-reliability evidence tier (source level L4). While ECE doesn't top aggregate calibration metrics like Expected Calibration Error or Brier score against the strongest baselines, it demonstrates a clear selective-prediction trade-off: near-perfect precision on answered claims by strategically abstaining when evidence is epistemically weak. The code is available on GitHub.