GPT-4.1, Claude Sonnet 4.6 match Gemini Embedding 2 on hard-negative image retrieval
Frontier LLMs rival Google's native multimodal embeddings on Flickr30k, no fine-tuning needed.
A new arXiv paper from Archan Dutta and Vyanktesh Kanungo (arXiv:2608.11343) pits frontier LLMs against Google's Gemini Embedding 2, the first natively multimodal embedding model that maps text, images, video, audio, and documents into a single shared space. Traditional multimodal retrieval relies on dual-encoder models trained with contrastive learning to align visual and textual representations. Gemini Embedding 2, released in March 2026, directly challenges that paradigm by producing joint embeddings across all modalities from the ground up.
The researchers evaluated both approaches on Flickr30k using hard-negative text-to-image retrieval, a notoriously difficult task requiring fine-grained discrimination between visually similar images. Surprisingly, zero-shot LLMs — specifically OpenAI's GPT-4.1 and Anthropic's Claude Sonnet 4.6 — matched the retrieval quality of Gemini Embedding 2 without any retrieval-specific training. However, the study reveals a critical deployment caveat: once embeddings are precomputed, native multimodal embeddings offer significantly lower latency for real-time applications. LLM-based rankers require per-query inference, making them slower at scale. This first-of-its-kind comparison suggests that frontier LLMs can act as flexible zero-shot rankers, but native embeddings remain the better choice for high-throughput, low-latency production systems.
- GPT-4.1 and Claude Sonnet 4.6 achieve retrieval quality on par with Google's Gemini Embedding 2 on Flickr30k hard-negative text-to-image tasks
- First direct comparison of zero-shot LLM rankers vs. natively multimodal embeddings — no fine-tuning required for LLMs
- Precomputed embeddings (Gemini Embedding 2) beat LLM-based ranking on latency, making them better for real-time search and recommendation
Why It Matters
For AI engineers, this means LLMs can plug into retrieval pipelines immediately, but latency-sensitive products still need dedicated embedding models for scale.