PyTorch adds cuDNN SDPA support for d=256, speeding up attention
New cuDNN backend in PyTorch now handles 256-dim heads with GPU acceleration.
Deep Dive
PyTorch merged a commit (PR #185553) adding cuDNN support for scaled dot-product attention (SDPA) with head dimension d=256. The change requires a cuDNN frontend upgrade.
Key Points
- PyTorch added cuDNN SDPA support for head dimension d=256 in PR #185553, filling a gap where only d=64, 128, or 512 were supported.
- The change unlocks NVIDIA cuDNN's flash attention for many transformer models that use 256-dim heads, such as BERT and GPT variants.
- Requires a cuDNN front-end upgrade, but promises faster training/inference and better GPU utilization for SDPA operations.
Why It Matters
Faster attention for 256-dim heads in PyTorch means cheaper training and inference for many production transformers.