SHIFT framework boosts implicit reasoning for LLM-based retrievers
New training method resolves retrieval-generation mismatch with self-reconstruction and fine-grained thinking.
LLM-based retrievers are critical for modern information retrieval systems. Current methods follow either a 'rewrite-then-retrieve' paradigm with explicit reasoning or implicit-reasoning approaches (e.g., GIRCSE, LaSER) that use soft tokens for efficiency. However, these implicit methods suffer from a mismatch between retrieval objective and generation objective, limiting their effectiveness on reasoning-intensive tasks.
To address this, Yuxiao Luo and colleagues introduce SHIFT (Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval). The framework first transforms LLMs into reasoning-efficient retrievers using residual projection and task-oriented bidirectional attention aggregation in latent space. It then alleviates the mismatch between contrastive learning and implicit reasoning through fine-grained next-token-prediction-based reconstruction. Extensive experiments on reasoning-intensive retrieval benchmarks show SHIFT consistently outperforms widely used retrievers, including GIRCSE and LaSER, demonstrating superior performance on complex queries that require deep understanding.
- Residual projection and bidirectional attention enable efficient implicit reasoning in latent space.
- Next-token-prediction reconstruction aligns contrastive learning with reasoning objectives.
- Outperforms GIRCSE and LaSER on multiple reasoning-intensive retrieval benchmarks.
Why It Matters
Enables more accurate retrieval for complex queries, improving AI systems' ability to reason over information.