New 'Few-class Fidelity' metric improves XAI evaluation for CNN classifiers

Convolutional Neural Networks (CNNs) power many real-world classification tasks, but their black-box nature demands explainable AI (XAI) to prevent bias and ensure trust. However, existing XAI evaluation metrics lack consensus and often rely on perturbation-based methods that may not represent real data distributions. In a new arXiv paper, researchers introduce Few-class Fidelity, a variation of fidelity-based XAI metrics designed for real-conditions applications with low class counts. The method generates in-distribution, uncertainty-provoking perturbations—optimized to stay within the data's true manifold—allowing a more accurate measurement of how faithfully an XAI explanation reflects the model's decision process. This is critical in domains like medical imaging, where only a few disease categories are present and perturbations must be clinically plausible.

The authors demonstrate Few-class Fidelity on both medical and natural imaging datasets, comparing it with human-centric object localization and segmentation metrics. Results show that the new metric correlates well with human annotations but also uncovers nuanced relationships between domain characteristics, data curation quality, and the choice of XAI technique. For example, it highlights how different data preprocessing steps can bias explanation faithfulness, guiding practitioners to select the right XAI method for their CNN model validation. The paper provides a robust evaluation framework that moves beyond synthetic perturbations, offering a practical tool for researchers and engineers deploying CNNs in sensitive fields like healthcare and autonomous systems where trustworthy explanations are paramount.