This AI Training Hack Distills ViT Models 25% Faster—And Adds a Stunning 12% Accuracy Boost
New curriculum lets small models skip layers to learn faster and better...
A team from multiple institutions (Jiaqi Zhang, Ashton Lee, Anthony Wong, John Zou, Sami BuGhanem, Randall Balestriero) introduced LEAP (Layer-skipping Efficiency via Adaptive Progression), a new training curriculum for Vision Transformer (ViT) feature-based knowledge distillation. The method directly addresses the teacher-student capacity gap in distillation by treating the teacher’s intermediate feature maps as a sequence of progressively more difficult targets. Rather than forcing a small student to match the full complexity of a large backbone (e.g., DINOv2) from the start, LEAP lets the student first learn foundational representations from early layers, then gradually tackle higher-level abstractions. This adaptive difficulty selection accelerates convergence and improves final performance.
Results are striking. On ImageNet-100, LEAP-distilled ViT-S achieves 90.1% accuracy — a +12.24% improvement over the baseline. On ImageNet-1K, retrieval tasks on Oxford and Paris datasets saw +3.84% and +7.75% gains respectively. Crucially, the curriculum enables early-stopping of teacher inference during initial training stages, yielding 25.1% savings in training FLOPs and 21% time savings on ImageNet-100. The approach scales across student model sizes and dataset scales. The code is publicly available, making this a practical recipe for edge deployment of vision AI.
- LEAP uses a curriculum of intermediate teacher features as progressive targets for small ViTs
- ViT-S distilled with LEAP hits 90.1% on ImageNet-100 (+12.24%), with 25.1% fewer FLOPs
- On ImageNet-1K retrieval, Oxford and Paris benchmarks improve +3.84% and +7.75% respectively
Why It Matters
Makes powerful vision models deployable on edge devices with less compute and better accuracy.