Video2Track turns real driving videos into steerable adversarial closed-track tests

Video2Track, developed by researchers including Mengjie Tian and colleagues, addresses a critical gap in automated driving validation: closed-track testing that relies on scripted maneuvers fails to capture the messy complexity of real roads. The framework takes raw driving videos and distills them into structured semantics using a vision-language model, then grounds those semantics on a closed-track topology library via retrieval-augmented generation (RAG). This identifies compatible map segments and interaction anchors, effectively translating what happened on public roads into a controlled test environment.

Once grounded, a dynamic module generates diverse multi-agent trajectories using a conditional diffusion model, while a Stackelberg game with a parameterized adversarial objective tunes the intensity of interactions. This allows engineers to push scenarios toward higher risk or adjust interaction styles — e.g., aggressive cut-ins versus defensive driving — without losing the realism of the original video. Closed-track experiments show Video2Track faithfully reproduces representative real-world interaction scenarios and produces executable variants with controllable risk levels. For safety teams, this offers a scalable path from passive dashcam footage to active, adversarial validation cases that stress an ADS where it matters most.