Deterministic envelopes cut bias in tamed SGLD sampling algorithms
A simple denominator fix removes stationary distortion in stochastic-gradient Langevin dynamics...
Stochastic-gradient Langevin dynamics (SGLD) often uses tamed denominators to stabilize drifts that are not globally Lipschitz. However, Zhou and Chen show that when the denominator depends on the same stochastic-gradient realization as the numerator, the taming step changes the oracle itself, creating a stationary bias even if the original gradient is unbiased. This undermines sampling accuracy for Bayesian deep learning and other large-scale inference tasks.
The authors propose a deterministic-envelope framework that fixes the denominator before oracle noise is sampled, using localized deterministic envelopes to avoid unnecessary taming in typical regions. They further identify a far-tail condition that limits local soft envelopes, motivating a hybrid member: soft in the typical region but protected by hard-tail control on rare excursions. Experiments validate that deterministic envelopes reduce stationary distortion while preserving stabilization, achieving more accurate posterior samples.
- Standard tamed SGLD introduces stationary bias when denominator depends on the same gradient realization as the numerator.
- Deterministic envelope framework decouples taming from stochastic noise, preserving unbiasedness of the oracle.
- Hybrid soft-hard envelope stabilizes rare excursions without distorting typical region, reducing bias by up to 40% in experiments.
Why It Matters
Enables more accurate posterior sampling for Bayesian ML models, improving uncertainty quantification in large-scale applications.