Lee et al. reveal why latent action models fail—and how to fix them

A new paper by Jung Min Lee, Taehyun Cho, Li Zhao, and Jungwoo Lee tackles a critical flaw in latent action models (LAMs), which attempt to learn action-like representations from unlabeled videos by compressing frame-to-frame changes. The core problem: real-world videos contain both the agent's own actions (endogenous state) and irrelevant changes like background clutter (exogenous state). These exogenous signals introduce noise that corrupts the learned latent actions.

The authors build an analytical framework by extending a linear LAM to explicitly model exogenous state. Their analysis yields two key insights: (1) minimizing the standard reconstruction objective inadvertently forces latent actions to encode exogenous information from future observations, and (2) learning in a representation space that isolates endogenous components is critical to mitigating interference. They also show that previously proposed auxiliary objectives, such as action-supervision, provably encourage latent actions to be consistent across different exogenous states. The findings are validated through experiments on both linear and nonlinear LAMs, offering a unified theoretical explanation for why latent action learning fails—and how common remedies actually work.