Claude Opus 4.6 and Gemini 3.1 Pro top new confidence calibration benchmark

A new research paper introduces ConfidenceBench, a benchmark designed to evaluate how well large language models (LLMs) calibrate their verbalized confidence—i.e., whether they can accurately express how likely they are to be correct. The study tested 15 frontier LLMs, including models from Anthropic, Google, and others, using 200 private multiple-choice questions across four categories: spatial reasoning, high-precision mathematics, word lookup, and unknowable questions. Calibration was measured using the Brier score, a proper scoring rule that incentivizes truthful probability reporting. The models were prompted to state their confidence without access to internal logits, making the framework applicable to both closed-source and open-source systems.

Claude Opus 4.6 and Gemini 3.1 Pro Preview tied for the best performance, each achieving a Brier score of 0.103—significantly better than the calibrated-random baseline of 0.1875. At the other extreme, Gemini 3.1 Flash-Lite scored 0.367, indicating severe miscalibration. A key finding is that accuracy and calibration are not aligned: the most accurate model is not necessarily the best-calibrated, and several models performed worse than the baseline despite reasonable accuracy. The results highlight that verbalized confidence calibration is a distinct and practically important dimension of LLM reliability, complementing standard accuracy-based evaluation.