MIT team uses VLMs to localize robots with OpenStreetMap semantics

A team of researchers from MIT (Ethan Fahnestock, Erick Fuentes, Philip R. Osteen, Nicholas Roy) has introduced a novel approach to city-scale cross-view localization that leverages the rich semantic information available in OpenStreetMap (OSM). Unlike prior methods that either ignore semantics or compress them into a few fixed classes, this work uses vision-language models (VLMs) to extract meaningful landmark descriptions from robot egocentric panoramas and then identify correspondences with overhead map features. The key insight is that VLMs can both recognize what is in front of the robot (e.g., 'a red brick building with a blue awning') and link that description to similar landmarks visible in satellite or map views.

However, naively using VLMs to propose all possible matches becomes computationally infeasible as the map grows (the system handles areas up to 628 km²). To solve this, the authors distill a lightweight neural matcher that learns from the VLM's outputs, enabling fast computation of correspondences for all entities in the map in a single forward pass. These correspondences are then used to form an observation likelihood, which is integrated over time via a Bayes filter to produce a time series of pose estimates. The method is trained exclusively on fair-weather data from Boston, yet demonstrates remarkable generalization across 11 environments spanning different cities, lighting conditions, and weather—including a snowstorm and nighttime runs in Boston.

To support further research, the team released a dataset of extracted semantics and evaluation trajectories. Their code is also publicly available. This work represents a significant step toward practical, generalizable robot localization in previously untraversed environments using widely available prior map data, potentially enabling autonomous navigation without requiring expensive pre-mapping or reliance on GPS.