Grasp-to-Dexterity pretraining boosts robot tool use success by 33%

Researchers at Carnegie Mellon University (Ying Yuan, Xinyu Liu, Sriram Krishna, David Held) have shown that large-scale dexterous grasp datasets—typically used only for grasp generation—can be repurposed as pretraining data for functional dexterous manipulation. Their hierarchical imitation learning framework combines high-level hand sub-goal prediction with a low-level goal-conditioned controller. The low-level controller is first pretrained on a 355k-trajectory dataset derived from existing grasp annotations, then fine-tuned on a small number of downstream task demonstrations. To benchmark this approach, they created DexCraft, a simulation environment with six articulated tool-use tasks requiring coordinated finger motion—such as operating scissors, pliers, and spray bottles.

In both simulation and real-world experiments, the pretrained controller significantly outperformed end-to-end diffusion policy baselines (e.g., DP3) and hierarchical policies trained from scratch. Notably, in real-world trials, the method improved full-task success by 33.3 percentage points over DP3. These results demonstrate that grasp datasets encode rich priors for contact-rich interactions and can serve as scalable pretraining resources for dexterous manipulation—moving beyond simple pick-and-place to functional, dynamic tool use.