The 50-Year-Old Math Behind Search Engines Finally Makes Sense
A quiet proof about how search ranks results could make AI answers smarter.
TF-IDF and BM25 are two of the most widely used methods for scoring query-document relevance — yet neither has had a standard probabilistic derivation justifying it as a statistical method within a unified framework. A new paper by Ivan Silajev addresses that gap by showing both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models. It treats the BM25 variant that includes the plus-1 correction in the IDF term, the one used in practice, and also discusses the original BM25 formulation without that correction. The resulting framework gives TF-IDF and BM25 a common theoretical basis, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods rather than only experimentally.
- TF-IDF and BM25 are the scoring formulas behind search bars and the 'look things up' step in AI chatbots — and the paper shows they are mathematically the same thing.
- BM25 dates back to the 1990s and still runs inside Lucene and Elasticsearch, used by thousands of websites and companies.
- There's no new product or speed upgrade yet — it's a five-page theory paper that hasn't been peer-reviewed, but it could guide better search and AI answers down the road.
Why It Matters
Better math behind search could mean more accurate results and fewer wrong AI answers for you.