ConvNN unifies CNNs and Transformers via k-nearest neighbor framework
Undergraduate thesis shows convolution and attention are just different neighbor selection strategies.
The traditional divide between Convolutional Neural Networks (CNNs) and Transformers is being challenged by a new framework called ConvNN, developed by Bowdoin College undergraduate Mingi Kang. ConvNN reveals that both convolution and self-attention are special cases of a single k-nearest neighbor (k-NN) aggregation process: convolution selects neighbors by spatial proximity, while self-attention selects by feature similarity. The framework can exactly replicate both operations—including depthwise convolution and sparse attention variants like KVT-attention—simply by changing the neighbor selection mechanism and weighting function.
ConvNN's key innovation is positioning CNNs and Transformers on a continuous spectrum rather than as distinct architectures. As a drop-in replacement for either layer type, it allows researchers to systematically interpolate between local and global aggregation. This could simplify architectural design, enable hybrid models that blend the strengths of both families, and open new avenues for building efficient vision models. While still an academic contribution, ConvNN has potential implications for future AI hardware and model efficiency.
- ConvNN exactly recovers standard and depthwise convolution by restricting neighbor selection to spatial coordinates.
- Exactly recovers self-attention and sparse variants (e.g., KVT-attention) using scaled dot-product similarity.
- Serves as a drop-in replacement for both convolution and attention layers, enabling exploration of the intermediate spectrum.
Why It Matters
ConvNN could simplify vision architecture design by proving CNNs and Transformers are not fundamentally different.