Statistically Guaranteed Hyperparameter Selection via Learn-then-Test

Hyperparameter selection is a critical but often ad hoc step in AI system deployment. Traditional methods such as grid search or Bayesian optimization provide no statistical assurance on reliability or safety. This monograph by Farzaneh and Simeone introduces a unified statistical framework centered on the Learn-then-Test (LTT) paradigm. LTT recasts hyperparameter tuning as a multiple hypothesis testing problem over candidate settings, allowing practitioners to select parameters that provably satisfy application-specific requirements—with explicit, finite-sample control of error probabilities.

The framework handles a wide range of reliability constraints, including bounds on average risk, quantile risk, and information-theoretic criteria. The statistical machinery is built from first principles, covering p-values, e-values, and concentration inequalities. Unlike black-box optimization, LTT delivers formal guarantees that selected hyperparameters will not exceed predefined error thresholds. This work is especially relevant for safety-critical AI systems (e.g., autonomous driving, medical diagnosis) where a wrong hyperparameter choice can have severe consequences. By moving from empirical tuning to provable guarantees, the authors offer a rigorous path toward trustworthy AI deployment.