Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance
A calibration trick using probability integral transform yields shorter, more reliable prediction intervals.
Conformal prediction offers distribution-free prediction intervals with guaranteed finite-sample marginal coverage, but achieving conditional validity and short interval lengths remains difficult, especially with heteroskedastic or skewed data. In a new arXiv preprint, researchers Zou, Zhu, and Nan introduce the Conformalized Percentile Interval (CPI), a calibration method that operates in the probability integral transform (PIT) space. By using a neural network to estimate the conditional cumulative distribution function (CDF) and then transforming responses to PIT values, the method leverages the asymptotic feature-independence of PIT values when the CDF estimator is accurate. This reduces feature-dependent miscoverage and improves conditional calibration.
The CPI method then calibrates percentiles directly on the empirical PIT distribution, making it robust even when the conditional CDF is imperfectly estimated. The authors prove finite-sample marginal coverage and show asymptotic conditional coverage under mild consistency conditions. Experiments on synthetic and real-world benchmarks demonstrate that CPI produces substantially shorter prediction intervals than existing conformal methods while maintaining better conditional calibration. This advance could significantly improve uncertainty quantification in high-stakes applications like healthcare, finance, and autonomous systems where both interval tightness and coverage guarantees are critical.
- CPI calibrates in PIT space using neural-network-estimated conditional CDFs, boosting conditional coverage without sacrificing marginal guarantees.
- Proven finite-sample marginal coverage and asymptotic conditional coverage under mild consistency conditions.
- Outperforms existing methods on diverse benchmarks, yielding significantly shorter intervals while improving conditional calibration.
Why It Matters
Tighter, conditionally valid prediction intervals enable safer decisions in AI systems under uncertainty, especially for heteroskedastic data.