Evolving Deep Learning Optimizers [R]

A team of researchers has developed a genetic algorithm framework that automatically discovers novel deep learning optimization algorithms. The approach encodes optimizers as genomes that specify combinations of primitive update terms—gradient, momentum, RMS normalization, Adam-style adaptive terms, and sign-based updates—along with hyperparameters and scheduling options. Through evolutionary search over 50 generations with a population of 50 individuals, evaluated across multiple vision tasks, the system identified a new optimizer that beats the widely used Adam optimizer.

The evolved optimizer combines sign-based gradient terms with adaptive moment estimation, but with key differences from Adam: lower momentum coefficients (β₁=0.86, β₂=0.94) and, notably, disabled bias correction. It also employs learning rate warmup and cosine decay scheduling. In aggregate fitness across several tasks, the evolved optimizer outperforms Adam by 2.6%, and on the CIFAR-10 dataset it achieves a 7.7% relative improvement. The results suggest that evolutionary search can uncover design principles that differ significantly from human-designed optimizers, potentially leading to more efficient training in computer vision and other domains.