Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment

A new preprint from researchers Spiros Baxevanakis, Platon Karageorgis, Ioannis Dravilas, and Konrad Szewczyk critically examines a popular 2024 solution for Vision Transformers. The original work by Darcet et al. identified that ViTs create confusing artifacts in their attention maps—the visualizations that show what parts of an image the model 'looks at.' The proposed fix was to add special, empty input tokens called 'registers,' which act as extra memory slots, helping the model store global information and clean up these maps.

This new study, a 26-page cross-architectural reassessment, puts that claim to the test across several major models including DINO, DINOv2, OpenCLIP, and DeiT3. While the researchers validated several of the original paper's key findings, their results crucially show that the necessity of registers is not universal. The effectiveness of the technique varies depending on the specific model architecture. Furthermore, the team extended the analysis to smaller model sizes and worked to untangle inconsistent terminology from the original paper, which is vital for accurately applying these concepts across the broader AI research community.