Resin et al. bridge calibration gap between classification and regression with new hierarchies

A new paper from Resin, Yang, and Gneiting tackles a fundamental problem in machine learning: how do we know if a model's probabilistic predictions are well-calibrated? The authors review, extend, and bridge calibration concepts across classification and regression tasks. They introduce a hierarchical framework that unifies disparate notions, from nominal classes to continuous outcomes. Practical implications include better evaluation of uncertainty in applications like medical diagnosis or financial forecasting.

Key contributions include the introduction of 'modal calibration' for nominal outcomes and a clear distinction between full, partial, and average calibration. Notably, the paper proves that double probability integral transform (PIT) calibration is logically independent of previously proposed concepts for discrete outcomes. The authors also generalize existing results for functionals like means, quantiles, and event probabilities. This work provides algorithmic tools for constructing instructive examples and counterexamples, helping practitioners understand when their models are truly calibrated.