TaskSense improves world models by filtering distractions

Researchers SM Mazharul Islam and Manfred Huber from the University of Texas at Arlington have introduced TaskSense, a novel framework designed to address a critical limitation in traditional world models for visual control: their tendency to prioritize task-irrelevant visual content due to the need for full observation reconstruction.

TaskSense tackles this by implementing a differentiable stochastic spatial attention mechanism conditioned on the previous latent state. This mechanism is further refined using an auxiliary inverse-dynamics objective, which guides the model to focus on control-relevant regions. Unlike conventional approaches that reconstruct the entire observation, TaskSense reconstructs only the attended regions, compelling the latent representations to retain task-relevant information while discarding distractions. The decoder integrates the sampled attention map to ensure consistent reconstruction despite the stochastic nature of attention. In evaluations, TaskSense matched DreamerV3’s performance on the DeepMind Control Suite but significantly outperformed it on the Distracting Control Suite, demonstrating superior robustness to visual distractions. Qualitative analysis confirmed that the learned attention effectively localizes control-relevant areas while suppressing irrelevant content.