Conformal Bayes tackles label shift: two calibration strategies compared
New research shows how to maintain prediction set validity when data distributions change.
A new paper from Seungjin Choi, presented at the 2nd Workshop on Epistemic Intelligence in Machine Learning (EIML@ICML 2026), tackles a fundamental problem in machine learning: how to maintain reliable prediction sets when the label distribution changes between training and deployment — a scenario called label shift. The work combines Bayesian posterior predictives with conformal calibration (conformal Bayes) to produce prediction sets that are both statistically valid and geometrically efficient.
Choi identifies two complementary strategies. Post-hoc calibration tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile — leaving the underlying parameter posterior unchanged. In-training adaptation, by contrast, tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region forms the prediction set. Two controlled experiments reveal that in an unbiased training regime both strategies achieve valid coverage equally. However, in a lead-optimization regime, in-training adaptation acts as a debiasing operator, reducing interval width while maintaining coverage — offering a clear practical advantage when training data is systematically biased.
- Two methods compared: post-hoc calibration (tilt predictive, not posterior) vs. in-training adaptation (tilt parameter posterior directly).
- In unbiased training, both achieve equal coverage; in lead-optimization, in-training reduces interval width at unchanged coverage.
- Importance-weighted conformal calibration is the shared mechanism for restoring target-domain coverage under label shift.
Why It Matters
Practical guidance for building robust AI systems that stay accurate when production data drifts away from training distributions.