SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

A team of researchers including Roxana Petcu, Evangelos Kanoulas, and Maarten de Rijke has developed SubSearch, a novel framework that addresses a fundamental challenge in AI: how to get large language models (LLMs) to perform reliable, multi-step reasoning for complex information retrieval tasks. Current approaches often rely on reinforcement learning based solely on final outcomes, or they require expensive human-annotated data to train separate reward models that judge each reasoning step. SubSearch innovates by shifting to "intrinsic process rewards"—internally-derived signals that directly optimize the LLM's generator to plan high-quality reasoning paths, eliminating the need for any external supervision.

Experiments conducted across seven established benchmarks demonstrate that rewarding these intermediate reasoning steps leads to significantly more robust and reliable reasoning traces, particularly in question-answering (QA) and multi-hop QA datasets. This represents a move toward more autonomous, information-intensive reasoning. The practical impact is substantial: SubSearch can help build AI agents that are better at integrating with tools like search engines to answer complex, multi-faceted queries. Furthermore, it provides a more data-efficient pathway compared to supervised process modeling, which could accelerate the development of capable reasoning systems without the bottleneck of human annotation.