New AI Model Lets Robots Discover Objects and Actions from Interaction Effects

Traditional robotic manipulation planning often relies on visually categorizing objects—a method that fails when objects look similar but behave differently (e.g., a mug vs. a cup). To overcome this, researchers introduce a model that jointly discovers high-level manipulation primitives and object categories by predicting the effects of interactions. The system uses a binary bottleneck layer trained on random interaction data to predict multi-modal outcomes, including object motion, contact feedback, and force feedback. This effect-driven approach allows robots to abstract continuous sensorimotor streams into discrete symbols for both actions and objects, enabling more nuanced understanding of the world.

The model leverages a discrete planning method that uses intermediate steps in predicted effect trajectories to enable partial action executions, giving precise low-level control. Crucially, it generalizes to novel objects in a few-shot manner: by comparing a small number of interaction effects with the predicted effects of learned object symbols, the robot categorizes new objects based on behavior rather than appearance. In tabletop repositioning and stacking experiments, this effect-driven planning outperforms both a state-of-the-art method and a vision-based alternative in precision across both seen and unseen objects. The work, detailed in arXiv:2607.00031, marks a shift toward robots that learn through doing—understanding objects by how they respond to actions, not just how they look.