GMAKE framework unifies Adam optimization with trust-region moments

Adaptive moment estimation optimizers like Adam are the default for training large neural networks, but their inner workings remain a patchwork of heuristics. A new paper by Oluwasegun A. Somefun proposes a trust-region framework that explains these mechanisms from first principles. The core idea is to constrain each weight update within a trust region defined by a moment constraint of order p, where p ranges from 2 (variance) to 4 (kurtosis). This yields GMAKE, a generalized family of learning-rate mechanisms that unifies second-moment estimation, normalized p-th moment estimation, learning-rate scheduling, momentum as spectral lowpass filtering, and even operator-level spectral normalization under one mathematical umbrella.

Somefun validates GMAKE on a GPT2-124M model trained on FineWeb-Edu and TinyStories datasets. The results reveal a clear trade-off: when trust-region constraints are weak, the fourth-moment (kurtosis-based) realization delivers its largest benefit, while stronger constraints make the classic second-moment realization increasingly competitive, often achieving slightly lower validation loss. This suggests that the optimal moment order depends on how aggressively you regularize the update step. By reframing optimizer design as a trust-region problem, GMAKE offers a principled roadmap for building custom optimizers that could outperform hand-tuned baselines across different architectures and data scales.