ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

A research team led by Yuhang Wang has released ADAS-TO, a groundbreaking multimodal dataset designed to study the critical moment when drivers take control from Advanced Driver Assistance Systems (ADAS). This first-of-its-kind public resource contains 15,659 synchronized 20-second clips pairing front-view video with vehicle CAN bus data, captured from 327 drivers across 22 different vehicle brands during real-world driving. The dataset specifically focuses on 'takeovers'—defined as transitions from ADAS ON to OFF—and categorizes them by trigger (brake, steer, gas, mixed, or system disengagement) and intent (planned 'Ego' vs. forced 'Non-ego' takeovers).

While most events occurred within safe kinematic margins, the team identified a long tail of 285 safety-critical cases. For these high-risk scenarios, researchers combined kinematic analysis with vision-language model (VLM) annotation to attribute hazards and link them to driver intervention patterns. The cross-modal analysis revealed distinct kinematic signatures across different traffic conditions, infrastructure issues, and adverse environments. The most significant finding: in 59.3% of critical cases, actionable visual cues were detectable in the video feed at least 3 seconds before the driver initiated the takeover. This suggests a substantial window for proactive, semantics-aware warning systems that could move beyond reactive, last-moment kinematic alerts.

The public release of ADAS-TO provides a crucial benchmark for developing and testing next-generation driver monitoring and assistance systems. By offering real-world, multimodal data on a vulnerable phase of automated driving, it enables researchers to build AI models that better understand the context leading to disengagement, potentially improving safety and smoothing the transition between human and machine control.