Human-like foveated vision makes transformers more efficient and robust
New FDT model beats DeiT-S accuracy while slashing compute by 34%.
Researchers from academia have unveiled the Foveated Dynamic Transformer (FDT), a novel vision transformer architecture that mimics the human visual system’s foveated sampling and eye movements. The model integrates a fixation module to identify key points and filter irrelevant information, and a foveation module to generate multi-scale embeddings. This biologically inspired design delivers a striking accuracy-efficiency trade-off: at a 50% fixation budget, FDT achieves 81.9% top-1 accuracy on ImageNet, outperforming the standard DeiT-S (80.9%) while reducing multiply-accumulate operations by 34.57%.
Beyond pure efficiency, FDT demonstrates inherent robustness to various types of noise and adversarial attacks, despite not being explicitly trained for such challenges. The authors attribute this resilience to the dynamic token selection process, which naturally suppresses corrupt or misleading patches. These results position FDT as a promising step toward more adaptive, resilient, and computationally efficient vision models for real-world deployment where both performance and robustness are critical.
- FDT achieves 81.9% accuracy on ImageNet at 50% fixation budget, beating DeiT-S (80.9%).
- Reduces multiply-accumulate operations by 34.57% compared to DeiT-S.
- Inherently robust to noise and adversarial attacks without explicit adversarial training.
Why It Matters
Biologically inspired token selection could make vision transformers faster, cheaper, and more resilient in production.