AutoSI automates statistical testing for ML feature selection

Researchers from Nagoya Institute of Technology and Japan’s National Institute of Advanced Industrial Science and Technology (AIST) introduced **AutoSI**, a framework that automates selective inference (SI) for machine learning algorithms. SI provides statistically valid p-values for hypotheses selected by algorithms, addressing the bias that arises when the same data is used for both selection and testing. Traditionally, implementing SI for new algorithms required experts to manually derive and implement selection events, limiting SI to narrow classes of algorithms.

AutoSI removes this barrier by automatically constructing the selection event from an algorithm’s individual operations, allowing users to write algorithms in standard NumPy-like code without manual derivations. It also broadens the class of selection events SI can handle, supporting any algorithm expressible through rational functions of the data (ratios of polynomials). The researchers proved that AutoSI’s p-values are exactly valid in finite samples and demonstrated its effectiveness on three feature-selection methods, including Lasso with tuning parameter selected by cross-validated R², which was previously unsupported by exact SI frameworks. Experiments on synthetic and real datasets showed that AutoSI controls the type I error rate at the nominal level while retaining high statistical power.