Robust Conformalized Selection (RCS) controls false discovery under label noise

Conformalized selection methods are widely used to pick top candidates from big datasets with rigorous uncertainty guarantees—think reliable labeling, drug discovery, or aligning large language models. However, existing approaches assume clean calibration data, which rarely exists in practice. A new paper on arXiv (2607.22985) from Chengyao Yu, Hongxin Wei, and Bingyi Jing exposes a critical flaw: under label contamination, standard conformal selection either fails to control the false discovery rate (FDR) or suffers severe power loss.

To solve this, the authors propose Robust Conformalized Selection (RCS), a unified framework that works for both selective classification and selecting candidates with responses above a threshold. The key insight: by conditioning on different classes separately, RCS transforms the messy label noise into a clean local covariate shift problem. It then uses a covariate-adjusted empirical-Bayes-type estimator to quantify the number of false selections. The paper proves asymptotic FDR control, power optimality, and robustness, and demonstrates effectiveness on simulated and real-world data. This matters because real applications almost always have noisy labels—RCS makes conformal selection practical again.