Gaussian MFVI overestimates predictive variance in ML models
New research reveals Mean Field Variational Inference misleads predictions by 2x in some cases
Deep Dive
A study found that Mean Field Variational Inference (MFVI) can overestimate predictive variance in Bayesian Linear Regression. While MFVI underestimates parameter variance, it overestimates predictive variance in directions where the training data concentrates. For test points drawn from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior.
Key Points
- MFVI underestimates parameter space variance but can overestimate predictive variance by 2x in data-rich directions
- Researchers demonstrated cases where MFVI predictive variance exceeds exact posterior for in-distribution test points
- The Cold Posterior Effect can correct this overestimation through temperature scaling
Why It Matters
This discovery impacts ML systems relying on uncertainty quantification, particularly in Bayesian deep learning and probabilistic modeling where accurate predictive variance is crucial.