FrequencyFormer: The 128x Edge ViT Compression That Rewrites Sensor Efficiency — And It All Happens in the Frequency Domain
Cut off-chip data movement by 128x with no retraining needed.
Deploying vision transformers (ViTs) on sensor-edge systems is bottlenecked by the energy and bandwidth needed to transmit high-dimensional image data from sensor to processor. Existing in-sensor computing methods offer only modest compression. FrequencyFormer tackles this by moving tokenization into the frequency domain at the sensor level. The pipeline includes a multi-scale DCT tokenizer that compresses a 224x224 image into compact frequency tokens, achieving up to 128x reduction in off-chip data volume with minimal accuracy loss. This is paired with a LUT-based near-sensor hardware implementation that uses fixed DCT coefficients for multiplier-free, energy- and area-efficient tokenization, plus a modified MIPI low-power communication architecture.
The pipeline achieves 28.8 TOPS/W, reduces communication energy by 230x, and lowers total sensor-side energy by 2.22x. Crucially, FrequencyFormer is a drop-in replacement for standard ViT patch embedding and remains compatible with pretrained backbones for classification, detection, and segmentation tasks. This makes it a scalable foundation for in-sensor ViT inference, enabling efficient AI on the edge without requiring model retraining or specialized silicon redesigns.
- Multi-scale DCT tokenizer compresses 224x224 images into frequency-domain tokens, achieving up to 128x off-chip data reduction.
- LUT-based near-sensor hardware uses fixed DCT coefficients for multiplier-free, energy-efficient tokenization.
- Pipeline delivers 28.8 TOPS/W, 230x lower communication energy, and 2.22x lower sensor-side energy, compatible with pretrained ViTs.
Why It Matters
Enables efficient vision transformer deployment on low-power edge devices without retraining, slashing data movement and energy.