Paragraphs improve vision-language AI 14+ points
A new study shows paragraph-level captions boost vision-language AI performance by 14+ points
A new arXiv paper systematically tests whether text granularity alone drives long-text retrieval. Using synthetic captions and quality-scored paragraphs generated with Qwen2-VL and Llama 3.2 Vision for 500K CC3M images, the authors fine-tune only the BLIP text encoder across 10 configurations. Paragraph-trained models match Long-CLIP-L on ShareGPT4V and beat it by over 14 points on DOCCI for image-to-text retrieval, with no architectural changes. They also show paragraph supervision supports long token sequences, while caption-only training degrades past 60 tokens; more diverse captions help short-caption retrieval with diminishing returns, and hard negatives hurt text-only fine-tuning. Evaluations span Flickr30k, COCO, ShareGPT4V, and DOCCI.
- Paragraph supervision boosts BLIP's image-to-text retrieval performance by 14+ points on DOCCI vs caption-only training
- Models trained with 500K synthetic paragraphs (generated by Qwen2-VL + Llama 3.2 Vision) outperform caption-only approaches beyond 60 tokens
- Fine-tuning only the text encoder with multi-sentence descriptions improves long-text retrieval without architectural changes
Why It Matters
Enables more nuanced image search from detailed descriptions, reducing reliance on short captions for vision-language AI applications.