CAFD uses VLMs to boost DNN fault detection by 18.3%

Fault detection in Deep Neural Networks has typically relied on model outputs or distance metrics, often requiring heavy computation. A new paper from Amin Abbasishahkoo, Mahboubeh Dadkhah, and Lionel Briand at the University of Ottawa proposes CAFD (Concept-Aware Fault Detection), which for the first time leverages Vision-Language Models (VLMs) to extract semantic concepts from input images. The method introduces a novel feature called Concept Failure Ratio (CFR) that quantifies how likely the presence of certain visual concepts is associated with DNN misclassifications. By combining CFR with traditional model-based signals and distance-based features, CAFD creates a richer, multi-source representation for detecting faults.

The empirical results are compelling: across three subject DNNs and datasets (including the large-scale ImageNet), CAFD consistently outperforms five state-of-the-art baselines across a wide range of constrained selection budgets. The average improvement in Fault Detection Rate (FDR) is 18.3%, with even larger gains in tighter budget scenarios. Crucially, CAFD achieves this without the substantial computational overhead typical of hybrid approaches, making it practical for real-world deployment. The work demonstrates that semantic information from VLMs can serve as a surprisingly effective signal for identifying when and why neural networks fail, opening new directions for debugging and quality assurance in AI systems.