A systematic literature Review for Transformer-based Software Vulnerability detection

A team of researchers led by Fiza Naseer has published a comprehensive systematic literature review (SLR) on transformer-based software vulnerability detection, analyzing 80 studies published between 2021 and 2025. The review follows Kitchenham's SLR guidelines and examines how transformer architectures—including encoder-only (e.g., BERT), decoder-only (e.g., GPT), and encoder-decoder models—are being applied to identify security flaws in source code, logs, and smart contracts. The study reveals that transformers outperform traditional machine learning and deep learning approaches due to their superior contextual modeling and representation learning capabilities.

The review identifies several critical technical challenges that remain unaddressed. Data imbalance across vulnerability types, lack of model interpretability, scalability limitations for large codebases, and poor generalization across programming languages are highlighted as key obstacles. The authors provide a consolidated resource for researchers and practitioners, mapping prevalent benchmarks, evaluation metrics, and reference models. This SLR serves as a foundational guide for developing more reliable, precise, and interpretable transformer-based vulnerability detection systems, particularly for high-stakes domains like healthcare, government, and finance.