Gaussian Joint Embeddings For Self-Supervised Representation Learning

Researcher Yongchao Huang has proposed a significant shift in self-supervised representation learning with the introduction of Gaussian Joint Embeddings (GJE) and its multi-modal counterpart, Gaussian Mixture Joint Embeddings (GMJE). The core innovation is moving away from the industry-standard deterministic architectures that align context and target views. Instead, GJE models the joint density of these representations, enabling closed-form conditional inference under an explicit probabilistic model. This approach directly addresses a key limitation: deterministic methods using squared-loss prediction tend to collapse toward conditional averages in genuinely multi-modal problems, losing crucial information about data variability.

The paper identifies and solves a critical optimization failure mode called the 'Mahalanobis Trace Trap,' developing several remedies including prototype-based GMJE, conditional Mixture Density Networks (GMJE-MDN), and a Sequential Monte Carlo memory bank. A notable theoretical contribution shows that standard contrastive learning can be interpreted as a degenerate, non-parametric limiting case of the broader GMJE framework. Experiments demonstrate that GMJE successfully recovers complex conditional structures in synthetic multi-modal tasks and learns competitive discriminative representations on standard vision benchmarks. Furthermore, the latent densities it defines are better suited for unconditional sampling compared to deterministic or unimodal baselines, opening new avenues for generative tasks within a representation learning paradigm.