SiMDex data mining lifts robot manipulation success rate to 61.1% from 47.7%

A new research paper from an international team (including authors from the University of Tokyo, Shenzhen University, and industry labs) presents SiMDex, a similarity-based data mining framework for cross-embodiment dexterous manipulation. As egocentric human videos explode in scale, a core question remains: which data actually helps robot manipulation? SiMDex reframes this as a recommendation problem. For each robot demonstration, it runs a three-layer recall-ranking-re-ranking pipeline over a massive pool of ~32M egocentric human samples, extracting task-relevant subsets in a morphology-agnostic action space. This means the framework works with any vision-language-action (VLA) model without retraining or architectural changes.

Against a strong baseline trained with the same amount of randomly sampled human data, SiMDex achieves a dramatic improvement: using only ~1.49M mined samples (under 5% of the pool), it raises the overall success rate from 47.7% to 61.1%—a 13.4-point boost. This demonstrates that careful curation outperforms indiscriminate mixing at equal data volumes. The framework is model-agnostic, operating purely on data selection, and can be applied to existing robot learning pipelines. For the robotics community, SiMDex offers a practical way to leverage large-scale human video datasets without drowning in noise, making billion-scale egocentric video pools far more useful for dexterous manipulation training.