HyenaND matches attention with native multi-dimensional subquadratic operators

The paper addresses a key limitation in subquadratic alternatives to attention: standard convolutions lack global receptive fields and input dependency, while recurrent models force multi-dimensional data (images, volumes, PDEs) into a one-dimensional scan order that violates spatial structure. The authors introduce HyenaND, a novel operator that acts directly on native multi-dimensional geometry through convolutions with implicitly parametrized global, input-dependent kernels. This design preserves locality and dimensionality while achieving global context without quadratic scaling.

HyenaND's CUDA implementation, nSubQ, fuses the FFT-convolution path to turn O(L log L) complexity into genuine wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines. Notably, hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids. The work demonstrates that native multi-dimensional subquadratic operators are viable and even superior to attention in certain settings, opening new avenues for efficient processing of high-dimensional data.