PRISM's multimodal mapping boosts rover navigation in rugged terrain

Robotic navigation in unstructured environments demands robust situational awareness to avoid hazards like steep slopes and rocky terrain. PRISM, developed by researchers from multiple institutions, addresses this by combining RGB, depth, and thermal imagery (RGB-D-T) into a unified perception system. The custom sensor suite captures aligned multimodal data, which is processed by OmniUnet—a novel vision transformer-based network designed specifically for semantic terrain segmentation. Unlike traditional RGB-only approaches, thermal imaging enhances differentiation of surface properties such as moisture or temperature, directly improving the reliability of traversability maps. This fusion enables the rover to classify terrain types more accurately, even in low-light or dusty conditions.

PRISM was validated using two newly annotated datasets (BASEPROD and LAENTIEC) and through physical field experiments on a real rover. Despite being deployed on a resource-constrained embedded computer, the system generates traversability maps efficiently enough to feed directly into the rover's Guidance, Navigation, and Control (GNC) subsystem for real-time autonomous navigation. This work demonstrates that multimodal sensor fusion with advanced transformers can run on limited hardware, unlocking safer exploration of planetary surfaces, disaster zones, or any unstructured terrain where traditional perception falls short.