AirTF fuses multi-modal tokens over the air for IoV, beating CNN baselines

In the Internet of Vehicles (IoV), transmitting high-dimensional multi-modal sensory data to edge servers for time-sensitive tasks faces severe spectrum bottlenecks. Existing schemes rely on convolutional neural networks (CNNs) with limited local receptive fields, which struggle to capture global context from distributed sensors. To address this, researchers from multiple institutions propose AirTF — a foundation model-driven over-the-air token fusion framework for task-oriented multi-modal token communications.

AirTF leverages vision transformer (ViT) encoders to extract globally contextualized semantic tokens from distributed heterogeneous sensors like RGB and infrared cameras. By concurrently transmitting these spatially aligned tokens over a shared wireless channel, the framework exploits the superposition property of the multiple access channel to inherently fuse complementary multi-modal semantics directly over the air. This mechanism significantly enhances spectral efficiency compared to orthogonal transmission. The integration of a pre-trained foundation model provides critical visual priors, effectively addressing the data-hungry nature of ViTs on limited, scenario-specific semantic segmentation datasets.

Experimental evaluations demonstrate that AirTF consistently outperforms orthogonal transmission and CNN-based fusion baselines across both AWGN and fading channels. Additional tests under a three-user setting, residual synchronization errors, and imperfect channel state information estimation further confirm its robustness. The source code will be made publicly available upon acceptance. This work points toward a new paradigm for efficient, real-time multi-modal fusion in bandwidth-constrained environments like IoV.