Conformal prediction gets coverage guarantees under distribution shift
New pseudo-calibration method keeps prediction sets reliable when data changes
Conformal prediction (CP) provides reliable prediction sets under exchangeability, but that guarantee breaks when the test data distribution shifts. In a new paper from arXiv (cs.LG/2602.14913), researchers Siahkali, Verma, and Gupta tackle this with pseudo-calibration. They model shifts as bounded label-conditional covariate shifts and use tools from domain adaptation to derive a lower bound on target coverage in terms of source-domain loss and Wasserstein distance. To keep coverage above a prescribed level, they inflate the conformal threshold by a slack parameter.
Their key contribution is a source-tuned pseudo-calibration algorithm that adapts its pseudo-labels based on classifier uncertainty—interpolating between hard deterministic labels and fully randomized ones. Numerical experiments show that the theoretical bounds track actual behavior, and the algorithm significantly reduces coverage degradation under shift while maintaining nontrivial prediction set sizes. This work offers a principled way to deploy CP in real-world settings where distribution drift is inevitable.
- Derives lower bound for target coverage using Wasserstein distance and source-domain loss
- Introduces slack parameter to inflate conformal thresholds and maintain coverage
- Source-tuned algorithm interpolates between hard and randomized pseudo-labels based on classifier uncertainty
Why It Matters
Ensures AI predictions remain trustworthy even when real-world data drifts from training