TileMaxSim GPU kernel accelerates ColBERT scoring by 220x

Existing GPU implementations for MaxSim scoring in multi-vector retrieval models like ColBERT run at only 5-18% of peak HBM bandwidth because they materialize the full similarity matrix. TileMaxSim, developed by Ashutosh Sharma, closes this gap with three key innovations: (1) multi-query SRAM tiling that streams document embeddings through shared memory and accumulates per-query-token maxima in registers, reading each embedding from HBM exactly once; (2) dimension tiling that partitions the embedding dimension into 128-wide chunks, enabling scoring for dimensions >128 that overflow shared memory; and (3) fused product-quantization scoring via shared-memory lookup tables that reduce HBM I/O by up to 31x.

On NVIDIA H100 GPUs, TileMaxSim reaches 80.2% of peak HBM bandwidth and scores 82M documents per second (71.6M/s on real MS MARCO passages). This represents a 220x speedup over loop-based scoring, 6.5x over fused PyTorch, 6.6-8.5x over prior GPU implementations, and 469x over WARP's CPU engine. It preserves exact retrieval quality—rankings on MS MARCO and three BEIR benchmarks match reference MaxSim. As a drop-in replacement in ColBERTv2/PLAID, it cuts scoring latency for 100K candidates from 268ms to 1.2ms (98% lower end-to-end). The kernel maintains constant throughput from 100K to 500K documents, supports multi-GPU sharding, and works robustly across dimensions 64-768 with FP16, BF16, and FP32 precision.