TheProfessor boosts vision-language model distillation by 1.77 points using two teachers

A team of researchers (Algadhi, Alzuhair, Alkhulaif, Behzad) introduced TheProfessor, a novel multi-teacher unsupervised prompt distillation framework for vision-language models (VLMs). The approach builds on PromptKD (CVPR 2024), which previously used a single PromptSRC-finetuned ViT-L/14 teacher to distill knowledge into a lightweight ViT-B/16 student. TheProfessor extends this by adding a second zero-shot EVA-CLIP-L/14 teacher, whose logits are pre-computed per dataset. Two ensembling strategies were tested: simple equal-probability averaging and a confidence-weighted method that adaptively blends teacher outputs.

In a 12-run single-seed sweep across Caltech-101, DTD, UCF101, and EuroSAT, confidence-weighted ensembling raised average harmonic mean (HM) from 87.52 to 89.28 (+1.77 points), while equal averaging achieved 88.88 (+1.37). Gains varied by dataset: negligible on Caltech-101 (+0.16 HM for confidence weighting), modest on UCF101 (+0.62), and dramatic on domain-shifted EuroSAT (+5.78). The results confirm that multi-teacher distillation is most valuable when the second teacher contributes complementary supervision, particularly under domain shift where the finetuned teacher may overfit and the zero-shot teacher provides robustness.

TheProfessor demonstrates a practical path to compress large VLMs (e.g., CLIP) into smaller, efficient models without requiring labeled target data. By leveraging multiple pretrained teachers, the student model gains both domain-specific knowledge and general visual understanding. This technique could accelerate deployment of VLMs on edge devices or in scenarios where labeled data is scarce, especially for applications like satellite imagery (EuroSAT) or texture recognition (DTD) that exhibit significant domain shifts from training data.