Vector DB benchmark: FAISS hits 866 QPS, Weaviate tops recall
7 databases, 4M vectors, 15 metrics — which ANN system wins?
Researchers Ashen Rashmiks and Tiroshan Madushanka published a comprehensive benchmark on arXiv (2608.12812) evaluating seven popular vector database systems for approximate nearest neighbor (ANN) search. The study covers FAISS, Qdrant, Milvus, Weaviate, Chroma, pgvector, and LanceDB, testing them on six diverse datasets—from classical SIFT and GIST visual descriptors to transformer-based text embeddings like MS MARCO and GloVe—totaling over 4 million vectors with dimensionalities from 96 to 960.
Across 15 metrics spanning retrieval quality (Recall@K, MRR, NDCG@K), query performance (latency percentiles, QPS, cold-start), and resource use (build time, memory, storage), the results reveal clear trade-offs. FAISS, a library rather than a full database, delivers the highest single-node throughput at 866 QPS on SIFT1M. Weaviate provides the best out-of-the-box recall at over 99%, while Qdrant achieves the best median latency among full databases at 4.55ms. LanceDB sacrifices retrieval quality for substantially faster index construction. The authors provide system-selection guidelines and open-source their benchmarking framework for reproducible comparisons.
- FAISS leads raw throughput at 866 QPS but lacks operational database features like replication and filtering.
- Weaviate achieves >99% recall out-of-the-box, while Qdrant posts a 4.55ms median query latency.
- Benchmark covers 4M+ vectors across 6 datasets and 15 metrics; framework released as open source.
Why It Matters
Provides data-driven guidance for choosing vector databases in RAG and semantic search, saving teams from costly trial-and-error.