Evolution Strategies with replay overcome catastrophic forgetting in control tasks

A team of researchers from the University of Pisa and partner institutions, led by Nicola Pitzalis, has released a new paper on arXiv (2608.13600) examining how Evolution Strategies (ES) — a family of black-box optimization algorithms inspired by natural evolution — perform in continual control settings. In these scenarios, an agent must adapt to a sequence of changing tasks without catastrophically forgetting how to solve earlier ones. Using sequential MuJoCo locomotion benchmarks, the team compared naive ES against ES augmented with replay buffers, which store and replay past experiences to reinforce prior knowledge.

The results are clear: naive ES suffers from severe catastrophic forgetting, quickly losing proficiency on earlier tasks as it adapts to new ones. Adding replay dramatically improves retention and, in some cases, induces positive transfer — where learning a new task actually improves performance on previous ones. However, the researchers also found a trade-off: larger replay budgets reduce plasticity, meaning the agent's ability to quickly adapt to new tasks diminishes. This tension between stability and plasticity is a classic challenge in continual learning. Accepted for publication at GECCO 2026, the study provides practical evidence that ES, combined with replay, can effectively support lifelong adaptation in control systems, offering a simple yet powerful recipe for robotics and autonomous agents that must operate in changing environments.