PAC-Bayes quotient method cuts generalization gap 21% in overparameterized models

Overparameterized models often contain continuous symmetries where different parameter sets produce identical predictors. This redundancy inflates PAC-Bayes generalization certificates because standard priors and posteriors can differ in parameter space even when they represent the same predictor. Aladrah and Anselmi solve this by performing the PAC-Bayesian analysis on the quotient predictor space—pushing priors and posteriors to equivalence classes of parameters. This removes the nonnegative KL contribution caused solely by distribution differences among parameterizations of the same predictor. They further construct a canonical parameterization per predictor that accounts for the geometric volume of equivalent parameterizations, transforming a neutral reference prior into a data-independent prior reflecting the model's implicit bias.

In experiments on Fourier regression with Hadamard parameterization, the implicit-bias prior reduced mean quotient-space KL by 40.69% and the mean PAC-Bayes certificate by 21.40% using ordinary SGD without any explicit regularizer. The effect was smaller but present in Query-Key attention, confirming its conditional nature based on prior scale. This work provides a principled way to tighten generalization bounds for overparameterized models—critical for deep learning theory and reliable deployment. The code and paper are available on arXiv (2607.18422).