Optimal Prediction-Augmented Algorithms for Testing Independence of Distributions

A team of researchers including Maryam Aliakbarpour, Alireza Azizi, and Ria Stevens has published a breakthrough paper titled 'Optimal Prediction-Augmented Algorithms for Testing Independence of Distributions' on arXiv. The work addresses a fundamental bottleneck in statistical inference: determining whether variables in a dataset are independent or related. Traditional independence testing requires prohibitively large sample sizes that scale polynomially with data complexity, making it expensive for real-world applications. The researchers' novel framework leverages the emerging paradigm of augmented distribution testing, where potentially untrustworthy predictive information from AI models can be incorporated to dramatically accelerate the process while maintaining statistical rigor.

The core innovation lies in designing testers that adapt their sample complexity based on prediction error. When auxiliary predictions are accurate, the algorithms achieve up to 90% reduction in required samples compared to conventional methods. Crucially, the testers remain robust even with poor predictions, preserving worst-case validity guarantees. The paper presents three main contributions: a bivariate tester for discrete distributions, a generalization to high-dimensional multivariate settings, and matching minimax lower bounds proving optimality. This work bridges statistical theory with practical machine learning, enabling more efficient analysis of relationships in complex datasets across genomics, finance, and social sciences.