GPUSparse delivers 235x faster exact sparse retrieval on GPUs
Exact SPLADE search on 8.8M passages at 787 QPS with 1.3ms latency
Ashutosh Sharma's new paper, GPUSparse, tackles the fundamental bottleneck of learned sparse retrieval inference: CPU-bound inverted index traversal. Sparse models like SPLADE offer interpretability and exact-match advantages but are slow in production. GPUSparse reimagines scoring as a GPU-parallel scatter-add over an inverted index, using block-aligned, warp-coalesced posting lists and fused Triton kernels.
On the MS MARCO passage ranking benchmark (8.8 million passages) with real SPLADE embeddings, GPUSparse achieves exact scoring accuracy—matching Pyserini's CPU MRR@10 of 0.383 and Recall@1000≥0.999—while running 235x faster (1.27ms vs 298ms per query). Compared to Seismic, the fastest CPU sparse retrieval system that trades 25% recall for speed, GPUSparse preserves exactness and delivers 787 QPS throughput on the full collection. The system reaches 62.6% of H100 peak HBM bandwidth, highlighting a fundamental tradeoff between work-efficiency and hardware utilization. This work opens the door to real-time, interpretable search at web scale without sacrificing accuracy.
- GPUSparse achieves 235x speedup over Pyserini CPU on 8.8M passages (1.27ms vs 298ms per query) while matching exact SPLADE accuracy
- System processes 500-query batches at 787 QPS using fused Triton kernels and a parallel inverted index, reaching 62.6% H100 memory bandwidth
- Outperforms Seismic (fastest CPU sparse system) by preserving exact recall (Recall@1000≥0.999 vs 0.738) at comparable latency
Why It Matters
Real-time, interpretable search at web scale without trading accuracy for speed—enables production-grade sparse retrieval on GPUs.