Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving
A new framework transfers AI driving agents from CARLA simulation to real Ford vans with up to 90% success, requiring no real-world training.
A team of researchers has introduced Sim2Real-AD, a breakthrough framework designed to solve a core problem in robotics and AI: transferring intelligent driving agents trained purely in simulation to operate reliably on real-world vehicles. The system tackles the notorious 'sim-to-real' gap by decomposing the challenge into four modular components. A Geometric Observation Bridge converts the vehicle's real monocular camera feed into a bird's-eye-view format the simulator-trained policy understands. A Physics-Aware Action Mapping then translates the policy's abstract commands into platform-agnostic steering and acceleration controls.
Crucially, the framework employs a Two-Phase Progressive Training strategy within the simulator to stabilize learning before deployment. The final Real-time Deployment Pipeline integrates perception, policy inference, and safety monitoring for closed-loop operation. The team validated Sim2Real-AD by training a Vision-Language Model-guided Reinforcement Learning policy exclusively in the CARLA driving simulator and then performing a zero-shot transfer to a full-scale Ford E-Transit. The real-world vehicle successfully executed complex tasks like car-following, obstacle avoidance, and stop-sign interaction with success rates of 90%, 80%, and 75%, respectively, marking a significant step toward more efficient and scalable development of autonomous driving AI.
- Achieved 90% success rate in car-following on a real Ford E-Transit van using a policy trained only in the CARLA simulator.
- Employs a four-module system including a Geometric Observation Bridge and Physics-Aware Action Mapping for zero-shot transfer.
- Demonstrates a scalable path to deploy advanced VLM-guided RL agents without costly and dangerous real-world training data collection.
Why It Matters
This dramatically reduces the cost and risk of developing autonomous driving AI by enabling safe, efficient training entirely in simulation before real-world deployment.