ResWM: Residual-Action World Model for Visual RL

A research team led by Jseen Zhang, Gabriel Adineera, Jinzhou Tan, and Jinoh Kim has published a paper on arXiv introducing ResWM (Residual-Action World Model), a novel framework designed to overcome a fundamental instability in visual reinforcement learning (RL). Traditional model-based RL systems condition future predictions on absolute actions, which creates an unstable optimization problem because the optimal action distribution is task-dependent and unknown beforehand, often leading to oscillatory or inefficient control. ResWM reformulates this by shifting the control variable from absolute actions to residual actions—modeling only the incremental adjustments needed from the previous step. This design aligns with the inherent smoothness of real-world physical control, dramatically reduces the effective search space for the AI agent, and stabilizes long-horizon planning.

To complement this shift, the team also developed an Observation Difference Encoder that explicitly models changes between consecutive visual frames, creating compact latent dynamics that naturally pair with the residual-action concept. The framework integrates seamlessly into existing latent dynamics models like Dreamer with minimal modifications and no new hyperparameters. Both imagination rollouts (where the agent practices in its internal world model) and policy optimization occur in this residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical testing on the DeepMind Control Suite showed ResWM consistently outperforms strong baselines like Dreamer and TD-MPC in sample efficiency, final performance (asymptotic returns), and control smoothness.

The implications extend beyond benchmark scores. By producing more stable and energy-efficient action trajectories, ResWM addresses a critical requirement for deploying RL-trained policies on actual robotic hardware in the real world, where jerky, high-variance motions are impractical and unsafe. The research suggests that residual action modeling provides a simple yet powerful architectural principle for bridging advanced RL algorithms with the practical demands of robotics, potentially accelerating the development of robots that can learn complex tasks directly from visual input.