HALO: Robot memory system retrieves from 8 minutes of past experience

A team of researchers from UT Austin (Rutav Shah, Yisu Li, Femi Bello, Yuke Zhu, Roberto Martín-Martín) has introduced HALO, a visuomotor policy designed to give general-purpose robots robust memory for long-horizon tasks in partially observable environments like homes. The key challenge in such settings is that robots must recall diverse past information (where objects were placed, which tasks a human completed, when an appliance was turned on) while learning from offline imitation data. Directly using attention over long contexts can introduce spurious correlations between past information and actions, and errors accumulate over time due to model drift and cascading failures.

HALO addresses these issues with two novel components. First, it distills vision-language model (VLM) priors into the policy by generating memory-dependent question-answer pairs from demonstration trajectories and training jointly with a video question-answering objective, steering retrieval toward task-relevant information. Second, to reduce the impact of accumulated errors during closed-loop control, HALO uses sparse attention that restricts retrieval to only the most relevant parts of history. Together, these enable reliable long-horizon control, with experiments showing successful retrieval from up to eight minutes of past experience. The code and project details are available on the project website.