Distributional Value Estimation Without Target Networks for Robust Quality-Diversity

Researchers Behrad Koohy and Jamie Bayne have introduced QDHUAC, a novel Quality-Diversity Reinforcement Learning (QD-RL) algorithm that addresses the critical sample efficiency problem in evolutionary AI. Traditional QD algorithms excel at discovering diverse skill repertoires but often require tens of millions of environment steps for complex tasks like robotics locomotion. While recent RL advances using high Update-to-Data (UTD) ratios accelerate learning, they typically rely on computationally expensive target networks for stability—creating a bottleneck for resource-intensive QD applications where rapid population adaptation is essential.

QDHUAC breaks this dependency by combining target-free distributional critics with dominance-based selection, providing dense and low-variance gradient signals that enable stable high-UTD training. The algorithm specifically enhances Dominated Novelty Search, a QD approach that balances novelty and quality. In experiments on high-dimensional Brax environments—standard benchmarks for continuous control—QDHUAC achieved competitive coverage and fitness metrics while using an order of magnitude (approximately 10x) fewer samples than established baselines. This represents a major leap in sample efficiency for evolutionary RL.

The paper, accepted as a full publication at GECCO'26, positions this target-free, distributional approach as a key enabler for the next generation of sample-efficient evolutionary algorithms. By removing the computational overhead of target networks while maintaining training stability at high UTD ratios, QDHUAC makes sophisticated QD-RL more accessible and practical for real-world applications where data collection is expensive or time-consuming, such as robotics hardware training or complex simulation environments.