Carnegie Mellon team proves optimal inference with black-box ML predictions

Powerful machine learning models are increasingly used to make predictions, but statisticians have struggled with a fundamental question: how do you combine a black-box model's output with actual observed data to run valid hypothesis tests? A new paper from Carnegie Mellon University researchers — Lucas Kania, Abhinav Chakraborty, Edward Kennedy, Larry Wasserman, and Sivaraman Balakrishnan — titled 'Optimal Inference with Black-box Predictions' (arXiv:2608.10155) provides the first unifying theoretical framework to answer this question.

The team analyzed the high-dimensional Gaussian sequence model, a canonical setting in statistics where researchers estimate many means from noisy data. They characterized the exact information-theoretic limits of inference when predictions come from a black-box model, both when the model's accuracy is known (e.g., from validation data) and when it is unknown but predictions are orthogonal. Building on these characterizations, they constructed practical hypothesis tests that automatically adapt to the unknown accuracy of the predictor — while still achieving near-optimal power when predictions are strongly aligned with the true signal. This resolves a gap in the field: earlier methods were either heuristic or only optimal under restrictive assumptions.

For data scientists, this means you can now rigorously quantify how much statistical power you gain by incorporating any ML prediction into your analysis, and use tests that remain valid even if the prediction is poor. The theory covers both known and unknown accuracy scenarios, enabling safe, efficient inference in applications like biomedical studies, A/B testing with covariate predictions, and any high-dimensional setting where black-box models are available as side information. The paper is available with full proofs on arXiv, and code links are expected to follow.