New AI Lets Drones Understand Your Spoken Directions
Soon you can tell a drone where to go in plain English—no piloting skills needed.
Aerial vision-and-language navigation (VLN) agents are typically trained on detail-rich, trajectory-aligned commands, whereas users issue short, intent-driven instructions; on a frozen OpenFly navigator, this \emph{instruction gap} drops success rate (SR) from $31.03\%$ to $11.33\%$. To scale translator training, we prompt a language model with human-written style examples to convert original commands into paired, intent-centered Weak commands, which yield $15.27\%$ SR.
We introduce the \textbf{Trajectory-Grounded Instruction Translator (TGIT)}, a front-end that keeps the navigator frozen and translates Weak inputs into agent-executable commands by learning from its trajectory outcomes. The resulting Weak-trained translator raises Weak-input SR to $37.93\%$ and transfers zero-shot to real human instructions ($11.33\%{\rightarrow}32.51\%$); it also improves held-out OpenFly ($4.95\%{\rightarrow}20.79\%$) and yields recovery on CityNav and AirVLN.
- The AI translates spoken instructions into drone flight paths, so you can control a drone by talking to it.
- It works with existing drones without modification, making it cheap and easy to adopt.
- So far, it's only been tested in computer simulations, not on real drones.
Why It Matters
This could let anyone use drones for tasks like delivery or farming, without special training.