Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning
Researchers solve the exploration-exploitation tradeoff in multi-agent systems with dynamic budget allocation.
Cooperative multi-agent reinforcement learning (MARL) faces a fundamental challenge: agents must explore vast state-action spaces to discover rare joint strategies, but too much exploration noise can overwhelm task rewards and cause coordination collapse. A new paper from Seoul National University (arXiv:2605.01865) introduces a quality-aware framework that dynamically adjusts exploration intensity both globally and per agent. The authors propose a Return-Conditioned Sigmoid schedule (RCB) that adapts the global exploration bonus β based on training progress, and a Reward Signal Quality (RSQ) metric that measures the signal-to-noise ratio of each agent's intrinsic reward. The key insight: agents receiving noisy intrinsic rewards should explore less aggressively. The framework uses Successor Distance (SD), a quasimetric intrinsic reward, to naturally produce distinguishable per-agent signal quality with convergence guarantees.
On seven cooperative benchmarks including MPE, SMAX, and MABrax, the method achieves top-tier returns across all environments, outperforming baselines that use fixed or uniform exploration budgets. The paper, submitted to Neurocomputing, provides theoretical guarantees on convergence and ordering preservation of agent exploration intensities. This work directly addresses the long-standing problem of exploration budget allocation in MARL, where previous approaches either used a single global intensity or ignored the varying reliability of agents' intrinsic signals. The practical implication: teams of robots or AI agents can now coordinate more efficiently by automatically allocating exploration effort to agents that can learn most from it, reducing wasted computation and avoiding catastrophic coordination failures.
- Proposes RCB (return-conditioned sigmoid schedule) for global exploration intensity control and RSQ (reward signal quality) metric for per-agent budget allocation.
- Uses Successor Distance (SD) quasimetric to naturally differentiate agent signal quality, with convergence and ordering preservation guarantees.
- Achieves top-tier returns on 7 benchmarks (MPE, SMAX, MABrax) by preventing both coordination collapse from excessive exploration and insufficient discovery of rare strategies.
Why It Matters
This method enables multi-agent systems to automatically focus exploration where it's most effective, improving coordination efficiency in swarms, robotics, and game AI.