CaRE protocol reveals temperature bias in masked diffusion model rankings

Masked diffusion language models (MDLMs) have become competitive with autoregressive models, but their evaluation standards are fragmented. Seven recent remasking papers used incompatible settings—varying step counts, metrics, and temperatures—making reported gains impossible to compare. To fix this, Yash Shah, Abhijit Chakraborty, and Vivek Gupta introduce CaRE (Compute-aware Remasking Evaluation Protocol). CaRE standardizes the actual number of function evaluations (NFE), enforces multi-metric reporting (including MAUVE), and explicitly controls stochasticity. The protocol was applied to 7 remasking strategies on LLaDA-8B-Base and Dream-7B-Base at 4 stochasticity levels and 3 step budgets on OpenWebText and LM1B datasets.

The results are striking. First, temperature explains the majority of MAUVE variance—meaning previous rankings may reflect hyperparameter choices rather than algorithmic merit. Second, compute-matched comparisons reverse several published strategy rankings, exposing evaluation artifacts. Third, informed remasking and stochastic unmasking are in tension: high-entropy remasking reduces MAUVE by 0.296 at 256 steps with unmask_temp=0.25 (p=0.020). A CaRE leaderboard covering 12 open-weight MDLMs (150M to 8B parameters) confirms these interactions hold across architectures and scales. The authors release the protocol, implementation, and leaderboard to make future MDLM claims truly comparable.