SMAC framework beats state-of-the-art in multimodal object tracking

Researchers from multiple institutions have introduced SMAC (Spatial-Modal Joint Modeling and Adaptive Representation Collapse), a new framework for multimodal multi-object tracking (MOT) that excels under challenging illumination conditions. The architecture addresses two core problems: insufficient joint modeling of spatial and modal features, and the inflexibility of fixed fusion strategies. SMAC uses a spatial-modal fusion backbone with two key modules: a Basic module that performs spatial feature extraction and modal interaction via decoupled 3D convolution, and a Mixed module that models nonlinear cross-modal correlations through amplitude-phase decomposition.

To enable adaptive fusion, SMAC introduces a representation collapse network containing a Distillation Prompt Guidance (DPG) module that generates dynamic modal weights under teacher supervision, and a Global Modal Difference Aggregation (GMDA) module that preserves discriminative information during collapse. Tested on the UniRTL dataset, SMAC achieves 63.31 HOTA and 79.21 MOTA on the RNT modality, outperforming existing state-of-the-art methods while maintaining favorable inference efficiency. The source code and pretrained models are publicly available, enabling further research and practical deployment in surveillance, autonomous driving, and robotics.