SemanticXR enables real-time object mapping on XR devices with 2% power overhead

Researchers from UIUC (Rahul Singh, Devdeep Ray, Connor Smith, Sarita Adve) have introduced SemanticXR, a novel device-cloud architecture that brings real-time, open-vocabulary semantic mapping to resource-constrained XR headsets and glasses for the first time. Existing semantic mapping systems are too compute-intensive for mobile devices and assume server-class resources. SemanticXR solves this by making semantically identifiable objects the fundamental unit of communication, execution, and memory across the device and cloud.

On the server side, object-level parallelism and geometry downsampling improve mapping latency by 2.2x while maintaining semantic quality. A depth-mapping co-design technique keeps upstream bandwidth under 2.5 Mbps. The device maintains an object-level sparse local map with incremental updates and prioritization, enabling query latency under 100 ms for up to 10,000 objects—even if the network drops. The system fits tens of thousands of objects within 500 MB of device memory, scales downstream bandwidth only with map changes (not total scene size), and adds just 2% power overhead during normal operation. This breakthrough makes grounded AI assistants and spatial object search practical on next-gen XR devices.