Reddit debate: Is capture-time annotation the missing link for robot learning?

A provocative Reddit post has sparked debate among robotics researchers: Is capture-time semantic annotation for robot trajectories still an unsolved problem? The user argues that raw teleoperation data—typically RGB video and joint states—structurally lacks critical information such as affordance (what an object can be used for), contact intent (where and how the robot should touch), and embodiment-specific kinematic context (how a particular robot's geometry affects motion). These cues are lost forever once the demonstration is recorded and cannot be reliably recovered post-hoc.

Current approaches either filter or clean data after collection, or rely on simulation to compensate for missing semantics. But neither method seems adequate for contact-rich manipulation tasks in unstructured, real-world environments. The post asks whether real-time supervision during acquisition—enriching the data stream as it's captured—could close this semantic gap. If not addressed, this may be a major bottleneck preventing robots from learning robust, generalizable skills from demonstration. The discussion highlights a growing need for novel sensor fusion or human-in-the-loop annotation tools at capture time.