QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning

A research team led by Yuanjun Li has introduced QSIM, a novel framework designed to solve a persistent problem in cooperative multi-agent reinforcement learning (MARL). Current state-of-the-art value decomposition (VD) methods suffer from systematic overestimation of Q-values because they use a max operator when calculating temporal-difference (TD) targets. This flaw is magnified in MARL due to the vast, combinatorial joint action space, often causing unstable training and suboptimal final policies. QSIM directly tackles this by reconstructing the TD target using a principle of action similarity, creating a more robust and accurate learning signal.

The technical innovation of QSIM lies in forming a similarity-weighted expectation over a structured set of actions near the greedy choice, instead of relying on a single maximum value. This allows the learning target to integrate information from diverse but behaviorally related actions, smoothing the update and effectively mitigating overestimation bias. The framework is designed to be plug-and-play, seamlessly integrating with existing VD methods like QMIX and VDN. Empirical results across standard MARL benchmarks show that QSIM consistently yields performance improvements of 15-30% and significantly enhances training stability compared to base algorithms. Its acceptance at the prestigious ICAPS 2026 conference underscores its potential impact on developing more reliable and capable multi-agent AI systems for complex, real-world coordination tasks.