SO-TA: New AI fusion method boosts robot manipulation success to 100%

Researchers from the National University of Singapore (Yue Feng, Weicheng Huang, I-Ming Chen) have developed Spacetime Optimal-Transport Attention (SO-TA), a novel fusion method for visuo-haptic imitation learning in contact-rich manipulation tasks. These tasks—such as tight-clearance peg-in-hole, connector mating, and surface wiping—are notoriously difficult because they involve discontinuous contact dynamics, partial observability, and strict safety constraints. Traditional approaches often rely on a single sensing modality (vision or force/torque), but SO-TA fuses three: vision (global context before contact), force/torque (interaction after contact), and proprioceptive pose (kinematic backbone).

The key innovation is replacing standard softmax-normalized patch attention with an entropy-regularized optimal transport (OT) alignment between force-pose-derived sub-queries and visual patches. This OT formulation imposes explicit marginal constraints that act as a structured inductive bias for contact-rich tasks, enabling conditioning-aware spatial selection stable across illumination changes, distractors, and partial occlusion. SO-TA is paired with a diffusion-based sequence policy that maps observation windows to pose-action chunks.

In real-robot evaluations with ~200 rollouts per condition, SO-TA achieved 100% success on tight peg-in-hole assembly (vs 93% for cross-attention) and 82.5% success under combined perturbations of lighting, distractors, and occlusion (vs 43.5% for a concatenation baseline). The OT-derived patch heatmaps and leave-one-out modality-influence ratios provide interpretable, phase-dependent diagnostics, showing which sensor stream dominates at each stage of the task.