DIVE embedding compression beats overfitting with self-limiting gradients

High-dimensional embeddings from large language models create storage and compute bottlenecks in vector search. Existing compression methods like Matryoshka-Adaptor, Search-Adaptor, and SMEC use lightweight residual adapters to reduce dimensions, but they overfit badly when labeled data is scarce—often performing worse than a frozen baseline. A new paper from Dongfang Zhao introduces DIVE, which tackles this problem with two key innovations: a self-limiting hinge-based triplet loss that produces zero gradient once the margin constraint is satisfied, limiting how much the pretrained embedding space can be perturbed; and a head-wise NT-Xent contrastive loss that treats multiple learned projections of each embedding as implicit views, providing dense self-supervised gradients that compensate for sparse triplet signals on small datasets.

Tested across six BEIR datasets (including TREC-COVID, NFCorpus, and SciFact), DIVE outperforms all three baseline adapters on every dataset and at every compression ratio evaluated. The approach is both simple and effective—the extra components add only 14 million parameters, and the full implementation is open-source. For practitioners running vector search at scale, this means being able to cut storage costs by compressing embeddings without sacrificing retrieval quality, even when fine-tuning data is limited. The self-limiting gradient mechanism also suggests a general principle: constraining updates can prevent catastrophic forgetting in adapter-based fine-tuning tasks beyond embedding compression.