Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes
New research mimics human eye movements to slash transformer network size and cost.
Researchers from Johns Hopkins University and Neurobaby Corporation have published a novel paper on arXiv titled 'Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes.' The work, led by Marc Estafanous, proposes a bio-inspired method to tackle one of the core limitations of transformer architectures: the quadratic computational cost of self-attention. Instead of processing an entire image sequence, their Saccade Attention Network (SAN) learns to identify and attend only to the most critical visual features, much like the human eye performs rapid 'saccades' between points of interest.
This approach uses transfer learning, where a smaller SAN is trained to predict the attention patterns of a large, pre-trained vision transformer. By doing so, it acts as a pre-processor, drastically reducing the input sequence length fed into the main model. The results are significant, demonstrating that calculations can be reduced by close to 80% while producing similar accuracy. This breakthrough points toward a future of leaner, faster, and more efficient computer vision models that are cheaper to train and deploy, potentially enabling more complex AI applications on resource-constrained devices.
- Mimics human saccadic eye movement to create sparse, efficient attention patterns.
- Uses transfer learning from a large model to train a small 'where to look' network.
- Achieves up to 80% reduction in computations while maintaining model performance.
Why It Matters
This could drastically lower the cost and energy use of training and running large vision AI models.