SegPAR attack beats black-box baselines in sparsity and efficiency

Adversarial attacks on semantic segmentation models have mostly focused on white-box or gradient-based methods, but real-world threats are often decision-based black-box, where attackers only see the model's output. Sparse decision-based attacks, which modify a minimal set of pixels, are especially practical yet understudied in segmentation. Existing approaches borrowed from image classification suffer severe query inefficiency because they accumulate changes image-wide, exhausting limited query budgets across high-dimensional pixel spaces.

SegPAR tackles this by reorienting exploration around semantic classes instead of pixels. Each class region is probed independently, dramatically reducing wasted queries. To avoid misleading gradients from standard decision rewards during pixel accumulation, SegPAR introduces a discrepancy reward that better reflects true segmentation changes. Extensive experiments show SegPAR outperforms black-box baselines in sparsity efficiency and mIoU reduction, while staying competitive with white-box sparse attacks. The code is open-sourced, allowing teams to test and defend against these practical threats.