IBM's lakehouse trick prunes Parquet files for 94x faster filtered vector search

IBM Research (Rakesh Jain, Thomas Griffin, Syed Zawad) proposes a new approach to filtered approximate nearest-neighbor (ANN) search in a disaggregated lakehouse, where data lives in Apache Iceberg tables over Parquet on object storage. Instead of bolting a separate filter step onto a specialized vector index, they embed an IVF (inverted file) index directly into each Parquet file's footer. The engine's existing file-pruning stack — partition pruning, zone maps, bitmap indexes — prunes data files first, then runs IVF only over the surviving files. This avoids loading all vectors and enables the query planner to compose pruning with per-file ANN retrieval natively.

The index is built distributed and non-destructively via a metadata-only Iceberg replace, so other engines still read the table. A rendezvous-hashed per-file cache keeps object-store latency in check. The payoff comes entirely from file pruning: on an 11.5M x 768 table, warm IVF search is ~32x faster than brute force at recall@10 ≥ 0.90 after pruning 355 of 444 data files. On 5M real IBM Granite embeddings, a filter arriving across a join prunes four of five region partitions, cutting query time from 14.7s to 157ms (~94x) once a region-partitioned layout is materialized. The authors also characterize when the method fails — it requires file-level locality on the filter column, and predicate pushdown is only safe with a provably pure partitioned column, not a merely sorted one.