New geometric framework explains contrastive learning failures under class imbalance

Researchers from CEA and Université Paris-Saclay have published a paper that redefines how we understand contrastive learning (CL). By interpreting weighted InfoNCE objectives as Distance Geometry Problems, they show that the weighting scheme dictates the target geometry of the learned embeddings. This unified framework allows exact characterization of optimal embeddings for several supervised and weakly supervised objectives. In supervised classification, both SupCon and its dense relaxation Soft SupCon collapse samples of the same class to a single prototype. However, while balanced SupCon recovers the classical regular simplex geometry, class imbalance introduces non-uniform inter-class similarities that depend on class sizes. Soft SupCon, by contrast, preserves the regular simplex regardless of imbalance, making it more robust for real-world datasets.

In continuous-label settings, the framework uncovers a critical mismatch: y-Aware Contrastive Learning generally cannot attain its entropic optimum unless the labels reside on a hypersphere. This reveals a fundamental incompatibility between Euclidean label weights and spherical latent similarity. By contrast, geometrically consistent choices such as Euclidean-Euclidean weighting or X-CLR are provably solvable and yield unique optimal embeddings. The authors conclude that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent. This work provides a principled, mathematically grounded approach for designing future contrastive objectives, offering clear guidance for practitioners who need to handle imbalanced data or continuous labels.