Document Optimization for Black-Box Retrieval via Reinforcement Learning
A new reinforcement learning technique makes small embedding models outperform larger, 6.5x more expensive ones.
A team from Stanford University and Hugging Face has published a novel research paper titled "Document Optimization for Black-Box Retrieval via Reinforcement Learning." The work, led by Omri Uzan, Ron Polonsky, Douwe Kiela, and Christopher Potts, reframes the classic technique of document expansion as an optimization problem. Their method fine-tunes a language model or vision-language model to transform documents offline, creating representations that better match the expected query distribution for a specific retriever. Crucially, it uses a reinforcement learning technique called GRPO (Group Relative Policy Optimization), where the retriever's own ranking improvements serve as the reward signal. This approach requires only black-box access to a retriever's ranking output, making it widely applicable across single-vector, multi-vector, and even traditional lexical retrievers.
The researchers evaluated their technique on challenging code retrieval and visual document retrieval (VDR) tasks. The results were striking: applying document optimization to OpenAI's smaller, cheaper text-embedding-3-small model significantly improved its nDCG5 scores—from 58.7 to 66.8 on code retrieval and from 53.3 to 57.6 on VDR. This optimized small model even slightly outperformed the much larger and 6.5 times more expensive text-embedding-3-large model, which scored 66.3 and 57.0 respectively. The method proves to be a powerful efficiency lever, enabling smaller, faster retrievers to match or beat larger ones.
When the retriever's internal weights are accessible, the paper shows that document optimization is often competitive with directly fine-tuning the retriever itself. Furthermore, combining both techniques yields the best overall performance. For instance, applying optimization to the Jina-ColBERT-V2 model boosted its VDR score from 55.8 to 63.3 and its code retrieval score from 48.6 to 61.8. This research provides a practical, cost-effective pathway to significantly enhance retrieval-augmented generation (RAG) systems and other search applications without expensive model upgrades or complex query-time processing.
- Uses GRPO reinforcement learning to optimize documents offline, requiring only black-box access to a retriever's ranking.
- Boosted OpenAI's text-embedding-3-small to beat the 6.5x more expensive -large model on code (66.8 vs 66.3 nDCG5) and VDR tasks.
- When combined with fine-tuning, improved Jina-ColBERT-V2's VDR score from 55.8 to 63.3 and code retrieval from 48.6 to 61.8.
Why It Matters
Enables companies to drastically improve search and RAG system performance using cheaper, smaller models, cutting inference costs by over 80%.