Post-ADC Inference fixes stats bias from active data collection
GP-UCB and TPE evaluations now yield valid p-values and confidence intervals
Deep Dive
A paper titled "Post-ADC Inference: Valid Inference After Active Data Collection" proposes a framework for valid statistical inference after active data collection (ADC) like SMBO. It corrects biases from adaptive sampling and post-hoc target selection using selective inference, producing valid p-values and confidence intervals with only noise assumptions—no black-box function or surrogate model assumptions needed. Empirical results show valid inference for data collected by GP-UCB and TPE.
Key Points
- Post-ADC Inference corrects bias from adaptive sampling (TPE, GP-UCB) and data-driven target selection
- Only assumes Gaussian observation noise—no assumptions on surrogate model or black-box function
- Provides valid p-values and confidence intervals, verified empirically on GP-UCB and TPE
Why It Matters
Enables reliable statistical inference from optimization data, unlocking trustworthy post-hoc analysis for AI/ML pipelines