VideoDB visual search infra beats commercial engines with 73% Recall@1

Most video-retrieval systems assume a bounded corpus and simply return file rankings or timestamps. But real-world agents watching cameras, streams, and archives face continuous, never-ending observations. In a new arXiv paper (2608.08075), Sankalp Nagaonkar and four co-authors argue this is an infrastructure problem, not a ranking problem. They introduce VideoDB (VDB), a production-ready data format that treats visual memory as coexisting scene spaces over shared source time, with three distinct layers: memory (everything retained), context (what is selected for a task), and evidence (the source intervals that ground results). VDB supports planned retrieval, stateful investigation, direct access, and grounded synthesis through a typed search surface.

The team benchmarked a general-purpose component pipeline against a commercial video-native foundation model across 9,800+ queries on four public datasets. The infrastructure approach won decisively at recall: 73.09 vs 65.75 at Recall@1, 83.39 vs 77.13 at Recall@3, and 91.20 vs 89.10 at Recall@10. The commercial engine only edged ahead at Recall@50 (96.42 vs 96.07). The key insight: retrieval quality over visual data is now governed more by system design—segmentation, sampling, embeddings, index layering—than by video-specific pretraining. VDB also makes live streams first-class sources and keeps every result linked to inspectable, playable evidence, a major advantage for auditability in real deployments.