New Double ML Method Tackles Distribution Shift and Selection Bias in Model Evaluation

Before deploying a prediction model in a new environment, understanding its likely performance is critical to preventing harm—especially when algorithms inform decisions. Two well-known culprits degrade model accuracy: covariate shift, where the distribution of input features changes between training and deployment, and selective labels, where outcomes are only observed for cases that received a particular historical action (e.g., a patient being tested). While each issue has received separate attention, their joint effect has been underexplored. In their new arXiv paper (2606.14506), researchers Annie Ulichney and Amanda Coston propose a double machine learning procedure that simultaneously handles both challenges. They derive a bias-corrected estimator using the influence function of the target risk, showing that it can evaluate an arbitrary black‑box prediction model under a general loss function. Their approach is identified under standard causal assumptions.

To validate their method, the team ran experiments using the eICU electronic health records database. The proposed estimator consistently tracked the true target risk more accurately than baselines that address only covariate shift or only selective labels, as well as combined plug‑in approaches. This work is particularly relevant for high‑stakes domains like healthcare, where models trained on one hospital’s patient population may face drastically different feature distributions and selective outcome recording due to clinical protocols. By offering a principled evaluation framework before deployment, Ulichney and Coston’s method could help practitioners avoid silent model failures. The full paper is available on arXiv under stat.ML and cs.LG.