GemNav uses frozen MLLM + LoRA for data-efficient robot navigation
Trained on just 8.7 hours of data, navigates 4 unseen environments zero-shot.
Typically, visual navigation policies require a dedicated visual encoder, a bespoke action head, and thousands of hours of cross-embodiment data. GemNav breaks that mold by adapting a frozen Multimodal Large Language Model (MLLM) using Low-Rank Adaptation (LoRA) on the language tower only. No auxiliary visual encoder, no continuous regression head—waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head. A soft-decoded auxiliary loss recovers the metric structure that pure cross-entropy training discards.
Trained on a single 8.7-hour open corpus (roughly three orders of magnitude smaller than competing datasets), GemNav transfers zero-shot to four physically distinct unseen environments: an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse. Over 20 real-world trials, the robot stops within 0.25–0.42m of the goal. Interestingly, conditioning on short image histories improves offline metrics but yields no additional robot benefit, suggesting a ceiling on temporal context once pretrained vision features are in place.
- Uses frozen Multimodal LLM with LoRA on language tower only—no dedicated visual encoder or continuous regression head
- Trained on just 8.7 hours of data, ~1,000× smaller than typical robot navigation datasets
- Zero-shot transfer to 4 unseen environments with 0.25–0.42m goal accuracy over 20 trials
Why It Matters
Massively reduces data and compute requirements for deploying foundation models in real-world robot navigation.