EMTC boosts multi-agent RL win rates by 24% in super-hard scenarios

Cooperative multi-agent reinforcement learning (MARL) suffers from reward sparsity and exploration bottlenecks. Existing episodic memory methods help by reusing high-return trajectories, but often trap agents in local optima due to unconstrained incentive distribution and semantic representation collapse. Researchers from the team of Zicheng Zhao et al. now introduce Episodic Memory Temporal Consistency (EMTC), a framework that robustly constructs and selectively leverages historical experiences. EMTC combines two components: a Temporally Consistent Semantic Embedder that integrates contrastive learning with time-conditioned state reconstruction to prevent representation collapse, and a Temporal Consistency Gating Mechanism that dynamically modulates episodic incentives based on temporal consistency error. This adaptive gate filters misleading signals from pseudo-successful trajectories, mitigating Q-value overestimation. The authors also provide theoretical guarantees, establishing a strict error bound linking observable temporal consistency error to trajectory optimality and representation quality.

Extensive evaluations on the SMAC and GRF benchmarks demonstrate EMTC's consistent outperformance over state-of-the-art baselines. Notably, compared to the strongest episodic baseline, EMTC achieves absolute win-rate improvements of up to 24% in super-hard SMAC scenarios and an average improvement of 28% across GRF tasks. These results are significant because they address a core limitation of episodic memory in MARL—trapping agents in local optima—by introducing temporal consistency as a principled filter. The framework is particularly relevant for applications requiring long-horizon cooperative decision-making, such as autonomous driving, robotics swarm coordination, and real-time strategy games.