Hadamard-CLIP: New method makes multimodal contrastive learning universal

In a new paper, Andrew Stuart (Caltech) and Florian Wolf analyze the expressive power of contrastive learning architectures, specifically CLIP-style models that dominate multimodal AI. They adopt a population-level, density-estimation view: each architecture defines a family of joint distributions, and they ask which distributions can be approximated arbitrarily well. For two modalities, they prove the classic two-tower CLIP is a universal approximator — meaning it can represent any joint distribution given enough capacity. That's the good news.

But for three or more modalities, the common practice of summing pairwise similarities is provably insufficient: these models cannot represent arbitrary joint distributions, though they can still match all pairwise conditionals. This explains a known gap in multimodal retrieval quality. The authors propose Hadamard-CLIP, which adds a single learned weight vector on top of the existing encoders to reweight pairwise terms. This tiny modification restores universal approximation for any number of modalities, with negligible computational cost and no change to the embedding retrieval pipeline. The result gives theorists a sharp characterization and gives practitioners a drop-in architectural upgrade for text, image, and audio models.