Audited Conformal Prediction (ACP) guarantees coverage under unknown data shift

Deploying machine learning models in production is challenging when the data at inference time differs from training—a problem known as distribution shift. Standard conformal prediction methods for uncertainty quantification assume a static environment, breaking down under shift. A new paper from Yanfei Zhou, Rizal Fathony, Nam H. Nguyen, and Matteo Sesia introduces Audited Conformal Prediction (ACP) to tackle this. ACP uses a small labeled dataset from the target population to train an auxiliary 'audit' model that predicts where the primary classifier is likely to make errors. By feeding the audit model’s outputs into the conformal prediction framework, ACP constructs prediction sets that maintain marginal coverage—the probability the true label is included—even under unknown shifts. Crucially, it also substantially improves conditional coverage, ensuring reliability for specific data subgroups.

The authors develop and analyze two complementary integration strategies. The first targets marginal coverage with enhanced conditional performance, while the second provides explicit group-conditional coverage guarantees for predefined subgroups—both backed by theoretical guarantees. Experiments on synthetic and real-world datasets validate ACP’s effectiveness, highlighting trade-offs between prediction set size and conditional coverage. For practitioners, ACP offers a practical way to deploy pretrained classifiers in dynamic environments without retraining. By adding a lightweight audit step, teams can monitor and correct for model blind spots, making AI systems more robust and trustworthy in production.