JEPA model detects driving complexity without any labels
Self-supervised JEPA spots risky scenarios like unprotected turns using prediction error alone.
A new self-supervised approach from Santosh Jaiswal uses a Joint Embedding Predictive Architecture (JEPA) to detect complex driving scenarios without any labeled data. Trained on structured agent state data from the nuPlan mini dataset, the model leverages temporal prediction error as a zero-shot proxy for scenario difficulty. In tests, it assigned significantly higher complexity scores to unprotected turns, crosswalk interactions, and pedestrian proximity, while lane-following and stationary traffic scored low. Four ablation experiments confirmed the signal source, and a downstream anomaly detection task yielded 0.512 Average Precision against a 0.436 chance baseline.
This work eliminates the need for human annotators, supervised classifiers, or hand-crafted rules—all of which require prior knowledge of what constitutes a difficult scenario. By showing that a minimal JEPA can autonomously learn complexity from raw agent state sequences, the technique could enable scalable safety analysis for autonomous driving. The method’s ability to flag rare but critical events without any labels makes it particularly valuable for mining large, unlabeled driving datasets, potentially reducing the cost and bias of conventional data curation pipelines.
- Achieves 0.512 Average Precision on anomaly detection vs 0.436 baseline using temporal prediction error as a zero-shot score.
- Trained on nuPlan mini dataset with structured agent state data—no labels needed during training or evaluation.
- Detects high-complexity scenarios: unprotected turns, crosswalk interactions, pedestrian proximity; low scores for lane-following and stationary traffic.
Why It Matters
Enables autonomous driving systems to identify risky scenarios in massive unlabeled datasets without costly human labeling.