AI system analyzes rail crossing safety using multi-modal data
New model achieves 0.757 F1 score identifying high-risk railway crossings from images and reports.
A research team led by Paimon Goulart and collaborators from multiple institutions has developed a multi-modal AI system for railway crossing safety analysis. Published on arXiv on July 1, 2026, the paper explores whether visual cues from images of crossings, combined with structured data like official accident reports, can produce robust safety estimates aligned with Federal Railroad Administration (FRA) standards. The proof-of-concept pipeline tackles challenges from data preparation to learning paradigms, ultimately routing inputs through a fine-tuned compact vision-language model (VLM).
The system demonstrates strong performance: it classifies crossings as HIGH-RISK or LOW-RISK with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and a correlation of 0.492. Qualitative results also align with domain-expert assessments. This work shows that combining visual data with structured accident history can enhance rail crossing safety evaluation, potentially reducing manual inspection effort and enabling faster prioritization of infrastructure upgrades.
- Proposes multi-modal pipeline fusing railway crossing images with structured accident reports.
- Achieves macro F1 of 0.757 for risk classification and RMSE 0.078 for FRA score estimation.
- Uses a routed fine-tuned compact VLM, producing assessments aligned with expert opinion.
Why It Matters
Scalable AI safety assessment could prioritize rail crossing upgrades, reducing accidents and informing infrastructure spending.