Researchers unveil adversarial decoys that fool ViT attention defenses
New attack patches misdirect attention-based defenses without affecting attacker's main goal.
Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks like adversarial patches, which can fool a model by altering a small region of an image. Recent test-time defenses attempt to mitigate this by detecting and suppressing tokens with abnormally high attention scores—the reasoning being that adversarial patches must attract substantial attention to influence the model's prediction. However, a new paper from researchers at the University of Pisa shows this assumption can be exploited.
The team introduces “adversarial decoys”: independently optimized patches that redirect the model's attention—and therefore any defense relying on attention scores—away from the actual adversarial region. The decoy is optimized separately from the main attack, making the method attack-agnostic and easily combinable with any existing adversarial patch technique. On ImageNet, decoys successfully suppress attention scores on the true adversarial patch while preserving most of the attack's misclassification ability. This work exposes a critical blind spot in attention-based defenses and calls for more robust detection methods.
- Adversarial decoys are independently optimized patches that redirect attention away from the true adversarial region, evading defenses that flag high-attention tokens.
- The method is attack-agnostic: the decoy is optimized separately from the misclassification attack, allowing it to work with any adversarial patch technique.
- Experiments on ImageNet across multiple ViT architectures show decoys preserve attack effectiveness while significantly reducing attention scores on the adversarial region.
Why It Matters
Attention-based defenses in ViTs are fundamentally flawed—attackers can now bypass them with a simple decoy patch.