OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies

A research team from the University of Pennsylvania and collaborating institutions has introduced OmniGuide, a novel framework designed to overcome the limitations of current Vision-Language-Action (VLA) models in robotics. While VLAs like RT-2 and GR00T excel at simple tasks, they struggle with complex operations requiring precise spatial understanding or manipulation in cluttered environments. OmniGuide addresses this by creating a unified system that can incorporate guidance from diverse external AI models—including 3D foundation models, semantic reasoning VLMs, and human pose estimators—and translate their outputs into actionable 3D guidance fields.

These guidance fields function as task-specific attractors and repellers in physical space, directly influencing the robot's action sampling process. For instance, a 3D model can provide an "attractor" field guiding a gripper to a specific handle, while a semantic VLM can create a "repeller" field to avoid fragile objects. The framework is flexible, allowing any model that can output a spatial energy function to contribute. Extensive experiments demonstrated that OmniGuide significantly boosted the performance of leading generalist robot policies, matching or surpassing prior methods designed for single guidance sources. This represents a major step toward more capable and reliable general-purpose robots that can safely perform intricate tasks in unstructured environments.