New method selects visual attributes from images, not LLM descriptions, boosting CLIP accuracy by 8%
LLM-generated attributes fail on real images—this method picks the right ones from the data itself.
A new paper proposes Distribution-Conditioned Attribute Selection for vision-language models. Instead of asking an LLM to describe class names, the method scores candidate attributes against actual images in CLIP’s embedding space. On ImageNet, class-name-free attribute prompts hit 23.8% accuracy (vs. 15.5% for LLM descriptors). With one image per class, it outperforms the prompt-tuning method CoOp by 3 points in under a minute (vs. 14 hours). The approach also provides a readable summary of dataset shifts.
- LLM-generated descriptors collapse to 15.5% accuracy on ImageNet when class names are omitted; DCAS achieves 23.8% by selecting attributes from actual images.
- With one image per class, DCAS outperforms prompt-tuning method CoOp by 3 accuracy points and trains in under 1 minute vs. CoOp's 14 hours.
- The selected attributes double as a readable summary of dataset shifts, enabling automatic description of distribution changes in words.
Why It Matters
Enables more robust zero-shot classification without relying on misleading LLM descriptors and automatically summarizes visual distribution shifts.