Evolution Strategies for Deep RL pretraining

A new research paper from a team of six authors, including Adrian Martínez and Ananya Gupta, investigates whether Evolution Strategies (ES) can serve as an effective pretraining step for Deep Reinforcement Learning (DRL). The study, titled 'Evolution Strategies for Deep RL pretraining', directly compares the two approaches across tasks of varying difficulty, from the simple game Flappy Bird to more complex environments like Atari's Breakout and the physics-based MuJoCo Walker. The core hypothesis was that ES—a more straightforward, derivative-free optimization method—could provide a computationally cheaper starting point to enhance the notoriously sample-inefficient and unstable training process of DRL algorithms.

The findings, however, challenge this optimistic view. The research concludes that Evolution Strategies do not consistently train faster than DRL methods. More critically, when used as a preliminary training phase, ES only yielded measurable benefits in the simplest test environment (Flappy Bird). For demanding tasks like Breakout and MuJoCo Walker, using ES for pretraining showed minimal or no improvement in either final performance, training efficiency, or training stability across different hyperparameter settings. This suggests that the hoped-for shortcut to better DRL agents via ES pretraining is largely ineffective for sophisticated, real-world problems.

The study, a 12-page project from an EE-568 Reinforcement Learning course, provides a valuable empirical check against common assumptions in AI research. By rigorously testing a proposed efficiency hack, it saves other researchers time and computational resources that might have been spent exploring this particular avenue. It reinforces that while ES has merits for certain problems, its role as a universal pretraining booster for advanced DRL remains unproven, directing focus back to improving core DRL algorithms themselves.