JEPA-MLM hybrid pre-training yields more semantic language embeddings
A single encoder with learnable balance between latent prediction and token masking.
Masked language modeling (MLM) has driven NLP since BERT, but its representations remain strongly tied to surface token identity rather than deeper semantics. Inspired by LeCun's Joint Embedding Predictive Architecture (JEPA) from vision and audio, this paper introduces a hybrid pre-training objective that combines a JEPA-style latent-space prediction loss with a standard MLM objective over a single shared encoder. A learnable scalar parameter continuously balances the two during training, and the model is pretrained on English Wikipedia with identical architecture and compute budget (NVIDIA H100) as a pure-MLM baseline.
Analysis across five GLUE benchmarks (SST-2, MRPC, MNLI, CoLA, STS-B) using four pooling strategies reveals consistent geometric differences. The hybrid encoder yields embeddings with uniformity below -0.16 versus -0.05 for MLM, exhibits richer spectral geometry under max pooling, and encodes less lexical surface information. Although linear-probe accuracy is similar, these geometric advantages suggest the JEPA objective reshapes latent space toward more semantic representations—a nuance standard metrics alone cannot capture.
- Combines JEPA latent-space prediction loss with MLM via a single learnable scalar parameter
- Hybrid embeddings achieve uniformity below -0.16 vs -0.05 for pure MLM on Wikipedia pretraining
- Maintains downstream GLUE accuracy while reducing surface-level lexical encoding
Why It Matters
Demonstrates that predictive objectives can produce deeper semantic representations in language models without sacrificing accuracy.