Adaptive Transformer sensor fusion cuts energy 90% for depth estimation on MCUs

A team of researchers (Crupi et al.) has developed an adaptive multimodal sensor fusion method for ultra-low-power depth estimation, targeting embedded systems like drones and IoT devices. Their architecture uses a lightweight recurrent Transformer (688k parameters) that sequentially processes inputs from a camera, ultrasound, and Time-of-Flight sensors. At each iteration, a confidence-based gating mechanism decides whether to continue computation with additional, more power-hungry sensors, enabling the system to trade off accuracy for energy on the fly.

Tested on a custom PCB with a GAP9 multicore System-on-Chip (sub-100 mW typical), the system achieves 2.44 mJ per frame — a 90% energy reduction compared to using all sensors simultaneously — while sacrificing only 4.8% accuracy (δ1 metric) on the NYUv2 dataset. Moreover, it uses 9× fewer parameters than MobileDepth yet retains competitive accuracy, and improves δ1 by 31.8% relative to a prior state-of-the-art model running on the same GAP9 hardware within the same power envelope (~400 mW). This shows that intelligent adaptive sensor fusion can dramatically extend battery life without sacrificing meaningful performance for real-world depth perception tasks.