New AI Lets You Control a Robot Swarm With Hand Gestures
Describe what you want, wave your hands, and 50 robots obey — no coding needed.
Auto-HSI is a method for generating personalized human-swarm interaction interfaces on demand. The objective: enable untrained operators to use natural language descriptions and gesture demonstrations to explain how they want robots to collectively behave in response to their gestures, with code automatically generated for personalized state machines that control the robots as desired. The developed prototype produces a personalized interface for centralized control using one- and two-handed gestures, letting a user teleoperate the robots' motion, formation shape, and shape deformation.
The gesture tracking and code generation components were tested against performance benchmarks. The full prototype was then tested in live operation experiments, in which real human operators centrally controlled 50 simulated robots in a physics-based simulator, under nominal and noisy conditions — teleoperating robots to score a goal, traverse a maze requiring shape deformation, and score two simultaneous goals by splitting into two groups.
A real human operator was also shown making live updates to their personalized Auto-HSI interface during operation, in simulation. Finally, live operation of real robots was demonstrated.
- One untrained operator used plain speech and hand gestures to command 50 robots at once, with zero coding.
- The AI wrote the robot control code automatically, and the operator could change their setup mid-task while robots were still moving.
- Tested on tasks like scoring goals, splitting into two teams, and squeezing through a maze that forced the group to reshape — still lab-stage work.
Why It Matters
Could let firefighters, farmers, and rescue crews command robot teams without needing a robotics engineer on site.