Researchers fix diffusion language models with spherical masking
New 'Spherical Soft-Masking' boosts diffusion language models by 2x on MAUVE scores
Researchers from a team including Lavanya Nigam and Aryan Sood have identified a critical flaw in how diffusion language models handle predictive feedback during training. Their paper titled *Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models* reveals that traditional linear interpolation (LERP) in embedding space is mathematically mismatched with the underlying geometry of mask and token embeddings in MDLMs, which exhibit a near-constant angular separation of about 73 degrees.
To address this, the team proposes Spherical Soft-Masking (S-SM), a drop-in replacement that computes predictions using the Fréchet mean on the hypersphere and blends with the mask direction via spherical linear interpolation (SLERP). Evaluated on a 169M-parameter MDLM checkpoint, S-SM delivers MAUVE score improvements of up to 2x over the vanilla baseline and 27.5–56.1% over TopK/LERP baselines across various sampling budgets. The method also consistently reduces generative perplexity by 16.9–19.6% without affecting output entropy or convergence speed.
- Spherical Soft-Masking (S-SM) replaces linear interpolation (LERP) with hyperspherical geometry and SLERP for more accurate predictive feedback in MDLMs
- S-SM achieves up to 2x MAUVE score improvement and 16.9–19.6% lower perplexity on a 169M-parameter MDLM
- The method maintains output entropy and convergence speed while outperforming TopK/LERP baselines by 27.5–56.1%
Why It Matters
Fixes a core training inefficiency in diffusion language models, enabling higher-quality text generation with measurable gains in coherence and perplexity.