Multi-Company Study Exposes Gaps in Autonomous Driving System Testing Practices

A new study published on arXiv, conducted by researchers from multiple institutions including the University of Rome Tor Vergata, TU Munich, and Chalmers University, interviewed experts from nine autonomous driving companies across six countries. The paper, titled "In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing," provides an in-depth look at current industrial practices, challenges, and future directions for ADS testing. Key findings reveal that most companies rely on scenario-based testing and X-in-the-loop (e.g., hardware-in-the-loop, vehicle-in-the-loop) approaches, supported by diverse simulation tools, metrics, and benchmarks. However, the study highlights major unresolved issues, including the difficulty of ensuring scenario realism, achieving comprehensive scenario coverage, maintaining simulation fidelity, and defining clear acceptance criteria. Experts also noted that existing testing standards are immature, and that different teams within the same organization often use inconsistent methodologies.

To address these problems, the researchers propose an evidence-centered closed-loop testing framework that systematically integrates testing data, scenario generation, and performance evaluation. Participants pointed to several promising solutions, such as leveraging AI and world models to generate more realistic and diverse test scenarios, as well as adopting end-to-end learning approaches to reduce reliance on hand-crafted rules. The study also envisions a future where ADS testing becomes more automated, data-driven, and transparent across the industry, with shared datasets and benchmarks to foster collaboration. This work serves as a comprehensive, industry-grounded overview that outlines clear directions for future research and practice, aiming to accelerate the safe deployment of autonomous vehicles.