AutoIndex: AI learns to rewrite documents for 43% better search recall
Your search index is a fixed preprocessing step? That’s about to change.
AutoIndex reframes document representation as an optimization target rather than a static preprocessing step. The framework searches over programs that can slice, normalize, reweight, or reorganize raw documents before they enter the retrieval system. At each iteration, agents diagnose failures of the current program and synthesize candidate updates, retaining only those that improve retrieval quality measured under the resulting index. This approach keeps the retriever (here BM25) fixed, proving that representation itself is a powerful lever.
Tested on CRUMB—a heterogeneous retrieval benchmark—AutoIndex improved recall@100 and nDCG@10 across all 8 tasks, with average gains of +8.4% and +8.3% respectively. The largest gains hit +30.5% Recall@100 and +43.6% nDCG@10, showing that learned transformations can dramatically boost even a simple bag-of-words retriever. The work suggests that the biggest retrieval wins may come from how we prepare documents, not from ever-larger models. Code is available for reproduction.
- AutoIndex treats document representation as a learned optimization, not a fixed preprocessing step.
- Achieved +8.4% average Recall@100 and +8.3% nDCG@10 gains over static BM25 baseline on CRUMB.
- Peak improvements of +30.5% Recall@100 and +43.6% nDCG@10 on individual tasks.
Why It Matters
Unlocks retrieval gains without changing the search algorithm—just by learning how to format documents.