GRU, LSTM, and Transformers fail at ADS detection under noise
Transformers and RNNs collapse under temporal jitter in ADS classification
A new arXiv paper evaluates the robustness of GRU, LSTM, and Transformer encoder models for classifying automated driving systems (ADS) like Tesla Autopilot and Comma Openpilot using vehicle telematics. The study, titled 'Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems,' found that all three models achieved high accuracy on clean data, with macro F1-scores of 0.92 (GRU), 0.90 (LSTM), and 0.93 (Transformer encoder). However, the models' performance degraded significantly under realistic telematics corruption scenarios.
The researchers introduced a modular robustness evaluation framework simulating five corruption families across five severity levels, including temporal jitter, additive noise, and event-level corruptions. While event-level corruptions had minimal impact (F1 ≥ 0.87 at L5), temporal jitter caused catastrophic performance drops, collapsing macro F1 to 0.44-0.50 across all models. This failure mode split underscores the vulnerability of current ADS classification models to temporal inconsistencies in real-world data streams.
- All three models (GRU, LSTM, Transformer encoder) achieved >0.90 F1 on clean ADS telematics data but collapsed to 0.44-0.50 F1 under temporal jitter
- The study introduced a robustness framework testing five corruption families at five severity levels, revealing critical failure modes in temporal data
- Findings highlight the need for more robust ADS monitoring systems for safety and regulatory compliance in real-world conditions
Why It Matters
Real-world ADS monitoring systems must handle noisy, corrupted data to ensure safety and compliance, but current models fail under temporal jitter.