H-Sets: Hessian-Guided Discovery of Set-Level Feature Interactions in Image Classifiers

Researchers at the University of Albany have introduced H-Sets, a novel two-stage framework for discovering and attributing higher-order feature interactions in image classifiers, accepted at CVPR 2026. Most existing feature attribution methods assign importance scores to individual pixels, overlooking how groups of features jointly influence model output—critical for semantic understanding in images. H-Sets addresses this by first detecting locally interacting pixel pairs via input Hessian matrices, then recursively merging them into semantically coherent sets using segmentation from Segment Anything (SAM) as a spatial prior. The second stage attributes each set with IDG-Vis, a set-level extension of Integrated Directional Gradients that integrates directional gradients along pixel-space paths and aggregates them with Harsanyi dividends. This targeted computational cost at the detection stage yields saliency maps that are sparser and more faithful than existing methods.

Evaluated across four architectures—VGG, ResNet, DenseNet, and MobileNet—on ImageNet and CUB datasets, H-Sets consistently produced more interpretable and faithful saliency maps compared to existing interaction-based methods, which are either too coarse (e.g., superpixel-only) or fail to satisfy core interpretability axioms. The framework's modular design allows the segmentation prior to be replaced with other methods, making it adaptable. By explicitly modeling set-level interactions, H-Sets provides a clearer window into how deep neural networks combine pixel groups into meaningful concepts, advancing the field of explainable AI for computer vision and potentially improving trust and debugging in real-world applications.