PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

A research team from Carnegie Mellon University and UC Berkeley has published a breakthrough paper on arXiv titled 'PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation.' The work addresses a critical bottleneck in robotics: teaching multi-fingered hands complex manipulation skills like those needed for household tasks. Traditional methods rely on either costly human demonstrations or reinforcement learning in simulation, but simulating realistic tactile feedback is computationally prohibitive. PTLD introduces a clever workaround by using privileged sensors in the real world to collect tactile data, which is then used to train a state estimator that can interpret touch, effectively bridging the sim-to-real gap without simulating touch itself.

The technical innovation lies in the distillation process. The team trains a policy in simulation using only proprioceptive data (joint positions). Separately, they collect real-world data from a robot equipped with tactile sensors. A 'teacher' model processes this rich tactile data to predict a latent state, which is then used to train a 'student' model that learns to estimate the same state from simulated proprioception alone. This distilled policy, when deployed on a real robot with tactile sensors, performs significantly better. On a benchmark in-hand rotation task, PTLD achieved a 182% improvement over a baseline with no tactile input, and a 57% improvement on a more challenging tactile reorientation task. This method paves the way for more data-efficient training of dexterous robots capable of nuanced, touch-based manipulation.