MSTF-Net: New UAV segmentation method fuses RGB and thermal video with 56% mIoU
UAVs can now segment scenes in real-time with a robust fusion of RGB and thermal data.
MSTF-Net tackles two key challenges in multi-spectral UAV video segmentation: the modal fusion dilemma (RGB and thermal features have significant discrepancies) and temporal variation (rapid motion causing misalignment across frames). The Modality Spatial Complementary Suppression and Enhancement (MSCSE) module generates unified instance queries via cross-modal attention and suppresses noise using residual-guiding. The Multi-scale Temporal Cross-modality Semantic Consistency (MTCSC) module adaptively adjusts the temporal receptive field based on frame distance, capturing both global context and fine local structure.
Extensive experiments on public RGB-T datasets demonstrate state-of-the-art performance, with 56.42% mIoU on MVSeg and 51.80% mIoU on CART. The method excels under challenging conditions like small targets, occlusion, and modality degradation. This enables more robust scene understanding for UAV applications in city planning, land use monitoring, traffic management, and crowd estimation.
- MSTF-Net achieves 56.42% mIoU on MVSeg and 51.80% on CART datasets.
- Uses MSCSE module for cross-modal attention and noise suppression.
- MTCSC module adapts temporal receptive field to handle rapid UAV motion.
Why It Matters
Robust multi-spectral segmentation empowers UAVs for safer autonomous navigation and reliable urban surveillance in all conditions.