Query Recycling lets 1.7B LLM match 7B search agents

A new paper from researchers at Universidade NOVA de Lisboa and Microsoft introduces 'query recycling', a technique that dramatically improves the efficiency of reinforcement learning for agentic search. The problem: GRPO-style algorithms, commonly used to train LLM search agents under outcome-only rewards, waste rollout cost on queries that produce zero variance—either all successes (too easy) or all failures (too hard). Previous work treated these as static and discarded or pre-filtered them.

The authors hypothesize and empirically validate that queries flip between zero-variance and signal-bearing states as the policy evolves during training. Their solution: return zero-variance groups to a mutable pool for future resampling, allowing the effective training distribution to co-evolve with the policy. Using this technique, a 1.7 billion parameter model trained purely on synthetic data achieves 66.0 average Pass@1 across seven multi-hop QA benchmarks, matching or surpassing systems with up to 7B parameters trained on benchmark-derived supervision. Analysis reveals that recycled queries supply roughly three quarters of the effective batch by the end of training, with contributions split between recovery from policy improvement and policy drift.