Dispatch-Aware Ragged Attention for Pruned Vision Transformers
Researchers fix a hidden bottleneck that made token pruning 60-90% slower than expected in real-world use.
A new research paper tackles a surprising performance roadblock in making Vision Transformers (ViTs) faster. Token pruning techniques like DynamicViT and EViT promise to slash computational costs by removing unimportant image patches. However, when these pruned, variable-length sequences are run through state-of-the-art optimized attention APIs—including FlashAttention-2's varlen and PyTorch's NestedTensor SDPA—the expected wall-clock speedups don't materialize. The researchers identified a 'dispatch-overhead bottleneck': for the short sequences typical of pruned ViTs (≤197 tokens), the host-side code path to launch the GPU kernel takes 60-90 microseconds, while the actual matrix math finishes in single-digit microseconds. This overhead completely overshadows the computational savings.
To solve this, the team built a lightweight, bidirectional attention kernel using Triton, a Python-like GPU programming language. Their 'dispatch-aware' design reduces the dispatch floor to roughly 40 microseconds, about 1.5x lower than FlashAttention-2 varlen. Integrated into a complete pack-attend-unpack pipeline, this system makes the theoretical benefits of pruning visible in practice. It achieves up to 2.24x higher end-to-end throughput compared to using padded sequences with standard PyTorch SDPA, consistently across four different pruning algorithms (Threshold-L2, DynamicViT, EViT, ATS) and DeiT model sizes. Crucially, it maintains bit-exact model predictions with less than 0.007 max absolute logit difference, ensuring no loss in classification accuracy.
- Identifies a 'dispatch-overhead bottleneck' where 60-90 us of host-side code negates the speed gains from token pruning in Vision Transformers.
- Presents a new Triton-based attention kernel that cuts dispatch time to ~40 us, about 1.5x faster than FlashAttention-2's varlen API.
- Delivers up to 2.24x end-to-end throughput gains over standard methods while preserving model accuracy across multiple pruning algorithms and model architectures.
Why It Matters
This work unlocks the real-world speed potential of pruned ViTs, making efficient computer vision models more practical for deployment in latency-sensitive applications.