Hierarchical BM25 slashes billion-doc lexical search to 4.4GB RAM

A flat BM25 index over one billion documents consumes roughly 400 GB of DRAM and takes 4-12 seconds per query when served from disk, making exact top-k lexical retrieval impractical for interactive applications. Hierarchical BM25, from researchers Umesh Deshpande and Swaminathan Sundararaman, sidesteps this by giving up exact ranking in favor of fixed memory and latency bounds. The system keeps a compact resident index of about 1,000 topic-balanced document groups, plus roughly 100 KB of global statistics. When a query arrives, it uses two signals to select which groups to visit: the total frequency of each query term within a group, and—for terms spread thinly across many groups—whether several informative terms co-occur in a single document within that group. Only the selected groups are searched exhaustively, and because scoring uses the same global statistics as a flat index, every returned score is identical to what the flat index would produce. The approximation is confined entirely to group selection.

The results are striking. The resident footprint is just 4.4 GB, independent of corpus size. In tests over one billion documents, sixteen-term queries returned in about 300 ms, delivering 4.7x to 5.6x the throughput of a flat multi-threaded index. With a warmed cache, the system sustains roughly 32 queries per second versus fewer than 3 for flat indexing—a 10-fold improvement. At a smaller 500K-document configuration, visiting only 5-10% of clusters recovers 0.83-0.92 of the exhaustive result score, showing strong recall-cost tradeoffs. The authors note that billion-scale recall measurements and direct comparisons against state-of-the-art exact algorithms like BlockMax-WAND remain open, but the practical implication is clear: lexical search at web scale no longer demands massive DRAM budgets, opening doors for on-premises, cost-effective retrieval systems.