AEM pretraining framework improves robot manipulation with temporal memory

A new paper from Yijing Zhou and colleagues presents AEM (Action-Effect Memory), a pretraining framework designed to improve robot manipulation by learning from temporal sequences of vision and action data. Traditional robot representation pretraining often focuses on single-frame visual encoding, which fails under partial observability where current observation alone is insufficient. AEM addresses this by modeling manipulation as an action-driven interaction process—it interleaves visual and action features and applies masked modeling to recover missing content from incomplete histories. This allows the system to learn action-conditioned state evolution. The output of the final vision token, encoded by a Mamba model, serves as a compact history representation, providing global context for decoding and downstream control while keeping inference efficient.

In extensive evaluations using both Diffusion Policy and Flow Policy backbones, AEM consistently improved manipulation performance across simulation and real-world settings. It outperformed baselines not only in clean scenes but also in cluttered and random environments, and on non-Markovian tasks where temporal context is critical. Ablation studies confirmed that history-aware pretraining surpasses single-frame pretraining and direct frame stacking, while also reducing inference latency and computational overhead. This work highlights the importance of temporal memory in robotics and offers a practical, efficient approach for integrating past actions into real-time control loops.