RMTL breaks long-horizon robot tasks with VLM-powered micro-task rewards

Reinforcement learning for robotic manipulation has long struggled with reward design: dense rewards are brittle to tune, and learning from human demonstrations is costly. A recent alternative uses pretrained vision-language models (VLMs) as zero-shot reward models, requiring only a text prompt. However, for long-horizon tasks with randomized initial conditions, a single global prompt yields a near-flat reward signal early in the trajectory, making it hard for agents to detect progress. Now, a team of researchers (Ateş, Kahraman, and Topal) introduces RMTL (Reinforced Micro-Task Learning) to solve this. RMTL decomposes a manipulation task into a small set of language-described micro-tasks and trains a hierarchical policy to switch between them. At each step, the agent receives a multi-view VLM reward computed using the prompt of the currently active micro-task, averaged across camera views to reduce occlusions. A reverse curriculum exposes the agent to harder initial conditions, and a PPO worker is first trained with a fixed distance-based rule that selects the active micro-task. This rule is later replaced by a learned hierarchical manager, turning phase selection into a fully learned policy.

RMTL was instantiated on the Fetch manipulation environment using just three short stage-specific prompts, without any additional prompt tuning. Experiments demonstrate that RMTL provides significantly more informative reward signals than single-prompt VLM rewards, enabling substantially faster learning. The decomposition into micro-tasks means the agent can detect reward changes even in early stages of a long sequence, overcoming the near-flat reward problem. By combining hierarchical learning, multi-view averaging, and reverse curriculum, RMTL makes language-guided RL more scalable for real-world robotic manipulation. The paper (arXiv:2606.26175) includes 16 pages and 11 figures, and is available under open license. This work suggests that carefully structured language prompts can replace expensive reward engineering, bringing us closer to robots that learn complex tasks from a simple verbal description.