Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures
A new study reveals that leading vision-language models for autonomous driving can be easily tricked by simple patch attacks.
A team of researchers from multiple institutions, including David Fernandez and Abolfazl Razi, has published a groundbreaking comparative analysis titled "Comparative Analysis of Patch Attack on VLM-Based Autonomous Driving Architectures." The paper, accepted at the 2025 IEEE Intelligent Vehicles Symposium, systematically evaluates the robustness of three prominent Vision-Language Model (VLM) architectures designed for autonomous driving: Dolphins, OmniDrive (Omni-L), and LeapVAD. Using a novel framework with black-box optimization and a process called 'semantic homogenization' to ensure a fair comparison, the team tested physically realizable adversarial patch attacks within the CARLA driving simulator.
The results are alarming for the future of AI-driven vehicles. The study reveals that all three tested VLM architectures possess severe vulnerabilities, suffering from sustained multi-frame failures when confronted with adversarial patches. These attacks caused critical degradation in the models' core object detection capabilities, a fundamental requirement for safe navigation. The analysis goes further to expose distinct patterns of vulnerability tied to each architecture's specific design, concluding that current VLM-based approaches are fundamentally inadequate at handling adversarial threats in safety-critical driving scenarios. This work shifts the conversation from theoretical AI robustness to a demonstrated, practical safety crisis for emerging autonomous systems.
- The study tested three VLM architectures (Dolphins, OmniDrive/Omni-L, LeapVAD) and found severe vulnerabilities in all.
- Using black-box optimization in CARLA sim, physically realizable patch attacks caused sustained multi-frame failures and object detection breakdowns.
- The research concludes current VLM designs are inadequate for safety-critical driving, exposing a major gap before real-world deployment.
Why It Matters
This exposes a foundational safety flaw in next-gen self-driving AI, pushing the industry to prioritize adversarial robustness before deployment.