Robot Hands Just Got Way Better at Picking Things Up

Teaching robot hands to manipulate objects with reinforcement learning is tricky: the random exploration that helps a robot discover finger-object contacts can also get in the way of precise control. In trajectory-guided settings like ViViDex, where RL refines hand-object trajectories from human video, the authors report that their baseline PPO still ran near its initial action noise after 5M steps. Their method, DexPolicy, makes the exploration scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer fixed.

They tested three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rose from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raised mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition.

The authors also report that PPO component screening favored noise control over the tested optimizer contraction, and that the selected PPO schedule yielded higher mean Target success than linear decay with the same endpoints on three tested objects. They conclude that training return, deterministic Target success, and tolerance to execution noise dissociate, so schedules should be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting.