MoDex lets robots grasp multiple objects sequentially with one hand
Robots can now pick up objects one after another without dropping any.
MoDex, developed by Haofei Lu and colleagues at KTH Royal Institute of Technology, tackles a long-standing challenge in robotic manipulation: sequentially grasping multiple objects with a single dexterous hand without dropping those already held. Traditional methods commit the entire hand’s degrees of freedom to one object, wasting the hand’s potential flexibility. MoDex uses a diffusion policy that predicts the next gripper pose from observations, conditioned on an opposition space (defining which fingers participate) and a point cloud. This lets the hand use only a subset of its fingers for each grasp, reserving the remaining fingers for subsequent objects.
To bridge the sim-to-real gap, MoDex is trained in two stages: first via imitation learning on expert demonstrations, then fine-tuned with reinforcement learning. Evaluated on a MuJoCo-simulated Franka Emika Panda robot with an Allegro Hand and on the real hardware, MoDex outperforms baselines by 2.92–17.92% in simulation and 6.67–17.78% in real-world tests. The work enables robots to perform tasks like picking up tools from a tray one by one — a critical skill for warehouse packing, surgical assistance, and household chores. Submitted to CoRL 2026, MoDex points toward more versatile, efficient robotic hands.
- MoDex uses a diffusion policy conditioned on opposition space and point clouds to sequentially grasp multiple objects without releasing held ones.
- Two-stage training: imitation learning on demonstrations followed by reinforcement learning fine-tuning, boosting success rates by up to 17.78% in real-world tests.
- Evaluated on a Franka Emika Panda robot with an Allegro Hand, outperforming learning-based baselines in both simulation and physical hardware.
Why It Matters
Robots gain multi-object dexterity for complex tasks like warehouse sorting and surgical tool handling.