Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecasting

A team from Northeastern University has published a major update to their work on predicting the movement of objects that disappear from view. Their paper, "Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecasting," expands the Out-of-Sight Trajectory Prediction (OOSTraj) task from just pedestrians to include both pedestrians and vehicles. This makes the research far more relevant to real-world applications like autonomous driving and robotics, where predicting the path of a car that moves behind a building is as critical as tracking a hidden pedestrian.

The core innovation is a new Vision-Positioning Denoising Module. Real-world sensors like LiDAR or radar provide noisy, incomplete data, especially when a target is occluded. This module cleverly uses known camera calibration parameters to establish a correspondence between visual data and positional sensor data. This allows the AI to effectively denoise the messy sensor signals in an unsupervised way, reconstructing a clean, predicted trajectory for the out-of-sight agent. It's a form of sensor fusion that compensates for the lack of direct visual cues.

Extensive testing on standard datasets like Vi-Fi and JRDB shows their method sets a new state-of-the-art benchmark, outperforming prior baselines and even classical approaches like Kalman filtering. The work, now published in the prestigious IEEE Transactions on Pattern Analysis and Machine Intelligence, is the first to use vision-positioning projection specifically for this denoising task, opening a new technical direction for making AI perception systems more robust and safe in unpredictable environments.