Researchers slash Caputo fractional gradient descent cost from quadratic to near-linear with memory compression

Fractional gradient descent (FGD) has long promised stability in ill-conditioned and nonconvex optimization problems by incorporating long-range memory through Caputo-type operators. However, its practical adoption has been stifled by computational costs that scale quadratically (O(T²)) with the number of iterations T, making it infeasible for large-scale deep learning. A new paper from researchers Hwanseo Lee, Junseo Lee, and Hyunju Kim directly tackles this bottleneck with two complementary memory compression mechanisms.

The first technique uses a sum-of-exponentials (SOE) approximation of the power-law kernel, converting the history-dependent convolution into efficient recursive updates that run in near constant per-step time. The second, more novel approach—dyadic hierarchical discrete convolution (DHDC)—compresses the gradient history through a multiscale aggregation strategy. Rather than treating these as purely numerical accelerations, the authors interpret them as controlled perturbations of the ideal Caputo operator. They prove that under standard μ-strong convexity and L-smoothness assumptions, the resulting method still exhibits monotone descent and linear convergence, provided the approximation error remains bounded.

This work effectively transforms FGD from a theoretical curiosity into a practical optimization tool. By reducing the memory cost from quadratic to near-linear (O(T log T) or even O(T) with SOE), the method becomes viable for training deep networks and solving ill-conditioned problems that benefit from long-range gradient dependencies. The perturbation-preserving analysis also gives practitioners clear error bounds to tune the trade-off between speed and fidelity.