Raubitzek et al. find fractional optimizers pair selectively with fractal activations

A sprawling new paper from Sebastian Raubitzek and colleagues at TU Wien pushes the boundaries of neural network optimization by merging two unconventional ideas: fractional-order optimization and fractal-based activation functions. Fractional optimizers generalize standard gradient descent by using fractional derivatives and memory effects, while fractal activations replace standard nonlinearities with self-similar Weierstrass- and Blancmange-type functions that operate at multiple scales. The 100+ page study systematically evaluates these approaches on classical benchmark surfaces—Ackley and Himmelblau—both clean and perturbed with Weierstrass-type noise, before moving to feed-forward networks trained on ten classification datasets.

The results reveal a nuanced picture rather than a universal win. Regularization-style fractional scaling, which applies memory in a controlled way, pairs well with certain fractal activations in practical network training. Grünwald–Letnikov memory, a classical fractional calculus discretization, proves most useful on perturbed surfaces where noise disrupts standard optimization. Notably, adaptive memory mechanisms outperform simple memory substitution in several cases, suggesting that the key isn't just adding memory—it's knowing when to apply it. The authors frame controlled fractional memory as a promising research direction, but caution that it doesn't replace conventional optimizers outright. For practitioners, the paper offers a roadmap for when and where these exotic techniques might actually help, without overselling them as a silver bullet.