GHOST framework helps robots generalize manipulation with hierarchical sub-goals

GHOST, developed by a team including Sriram Krishna and David Held, addresses a key challenge in robot learning: generalization beyond the training distribution. The framework factorizes visuomotor control into two levels: a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses using multi-view RGB-D observations, and a low-level goal-conditioned controller that executes embodiment-specific actions. To bridge the gap between 3D goals and image-based policies, the authors introduce a simple spatial interface that projects predicted goals into the image plane as end-effector heatmaps. Across multiple manipulation tasks, this hierarchical approach consistently improved both performance and robustness compared to a flat Diffusion Policy baseline.

The key innovation is how GHOST leverages the embodiment-agnostic nature of sub-goals. Because the high-level policy only cares about where the end-effector should go, it can be trained on human video demonstrations without needing action retargeting—a notoriously noisy step in imitation learning. The low-level policy remains trained purely on robot data, ensuring precise execution. This hierarchy enables rapid adaptation to novel objects and task variations using just a handful of human demos. Accepted at RSS 2026, GHOST represents a practical step toward robots that can learn from humans and generalize to new situations without massive retraining.