Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy

A team from Stanford and the University of Illinois has developed a novel framework called Real-to-Sim-to-Real Shared Autonomy that significantly improves the reliability of teleoperating robots for fine-grained, contact-rich tasks. The core innovation is using a minimal amount of real-world human data—less than five minutes—to train a simple k-nearest-neighbor (kNN) model that acts as a surrogate for the human operator within a simulation. This surrogate then enables the stable training of a corrective 'copilot' policy using model-free reinforcement learning (RL), something previously difficult due to the challenge of accurately modeling human behavior for simulation training.

The resulting AI copilot was deployed to assist real human operators in industry-relevant manipulation tasks such as nut threading, gear meshing, and peg insertion. A user study with sixteen participants demonstrated clear benefits: the system improved task success rates for novice operators and increased execution efficiency for experienced operators, outperforming both direct teleoperation and other shared-autonomy baselines. An additional, powerful side-effect is that the copilot-assisted teleoperation produces higher-quality demonstration data, which can then be used to more effectively train downstream robot imitation learning policies, creating a virtuous cycle of improvement.