New Study: Plain Search Still Beats Fancy AI Planning

When an AI has to answer a question using a huge pile of documents — a company handbook, a legal contract, a year of emails — it can't read everything at once. So engineers build shortcuts. One popular idea is "planning" (teaching the AI to decide in advance which pieces to read). Another is plain retrieval (simple search that pulls out the most relevant paragraphs). The flashier planning approach has been sold as the next step forward.

This study put them head-to-head. The researchers used 503 multiple-choice questions from a tough long-document test set, plus a smaller untouched set of 152 questions, and ran everything through a modest 7-billion-parameter AI model. At a generous reading budget, good old hybrid search scored 36.18% accuracy, keyword search scored 35.98%, and the best planning method managed only 34.19%. On the clean, never-seen test questions, plain search won by more than five points: 42.11% versus 36.84%. Even when they tried to be fair by reranking and routing the results, the gains were tiny — 0.40 points in one case, and a boost that vanished entirely at slightly larger budgets.

Why does this matter beyond a lab? Cost. Planning adds an extra AI step before every answer, which means more computing power, more delay, and a bigger bill. If a simpler search does the job equally well or better, that extra spending is largely wasted. It also matters for trust: teams building customer-support bots, legal research tools, or internal search have been told to invest in planning layers. This paper suggests the money is often better spent making the search itself stronger.

The catch: this is a five-page "negative result" paper, and negative results are about what didn't work in one specific setup. The training data overlapped with part of the test set, the authors admit, and only one model was used. So it isn't proof that planning never helps. Think of it as a warning label, not a funeral.