Sekimoto and Yasuda's round-trip kernel beats Gibbs sampling for RBM training
A single-step nonlocal transition kernel that outperforms deep tempering for RBM sampling...
Restricted Boltzmann Machines (RBMs) are energy-based models used for representation learning, but training them requires estimating intractable expectations via Markov chain Monte Carlo. Blocked Gibbs sampling (BGS), the standard approach, generates local moves that get trapped in high-energy regions, degrading learning. Deep tempering (DT) uses parallel tempering over a sequence of RBMs to escape local modes, but needs multiple transitions to travel the sequence and achieve nonlocal moves.
Kaiji Sekimoto and Muneki Yasuda introduce a nonlocal transition kernel defined on the RBM sequence from DT. The kernel has a round-trip structure that traverses the sequence back and forth in a single transition, enabling large jumps while preserving the target distribution. Numerical experiments show the proposed kernel produces nonlocal moves more frequently and achieves better sampling quality with fewer transitions than both BGS and DT. RBM learning based on this kernel is more stable and mitigates training failures observed with existing methods, offering a computationally efficient alternative for energy-based model training.
- Round-trip kernel enables nonlocal transitions in a single step over the deep tempering RBM sequence
- Achieves higher sampling quality with fewer transitions than blocked Gibbs sampling and deep tempering
- Produces more stable RBM learning and reduces training failures in numerical experiments
Why It Matters
Faster, stabler RBM training could scale energy-based models to harder distributions and improve generative AI efficiency.