New AI framework boosts image captioning accuracy by 9.6% without retraining
Researchers break zero-shot image captioning records with multi-agent consensus scoring
Researchers from Vietnam’s top computer science programs (including Duy Tran Thanh, Thien-Phuc Doan, and others) have published a breakthrough zero-shot image captioning framework called Adjudicated Captioning that slashes reliance on labeled training data while boosting accuracy.
The approach introduces a multi-agent scoring system that adds three frozen verification components to existing captioners: a stronger Retrieval Encoder at input, a Cross-Attention Verifier that re-ranks top-9 retrievals to top-5 between retrieval and decoding, and a learned Reranker (TriFuse + MemAttend) at the output beam. Crucially, these components use self-supervised Borda-consensus distillation to avoid paired image-caption labels. The result is a 117.6 CIDEr score on COCO Karpathy—up from 108.0 for the baseline IFCap captioner—with +7.8 points from architectural interventions alone and +1.8 from learned reranking. The method shows transferability to Flickr30k (+8.1 CIDEr) and NoCaps (+5.7 CIDEr) without retraining the captioner.
- Adjudicated Captioning achieves +9.6 CIDEr (117.6 vs 108.0) on COCO Karpathy without retraining the base captioner IFCap
- Uses three frozen scorers (Retrieval Encoder + Cross-Attention Verifier + learned Reranker) with self-supervised consensus distillation
- Transfers to other datasets: +8.1 CIDEr on Flickr30k and +5.7 on NoCaps without model fine-tuning
Why It Matters
Enables high-accuracy image captioning in zero-shot settings without costly labeled datasets, unlocking accessibility for low-resource languages and domains.