Researchers propose SAKE to boost text diffusion diversity

A team of researchers from multiple institutions has proposed a novel approach to improve diversity in text diffusion models while maintaining high fidelity. Their paper, titled 'Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance', introduces Semantic-Aware Kernel Entropy (SAKE), a training-free guidance method that dynamically adjusts the sampling distribution based on semantic interactions and token positions.

The core innovation lies in using order-2 Rényi entropy over a kernel Gram matrix to capture both cross-token semantic relationships and positional information. This entropy-based signal is linearized in the embedding space to create a tractable guidance mechanism that flattens the sampling distribution to encourage exploration during redundancy and sharpens it for fidelity when diverse outputs are needed. Experimental results demonstrate that SAKE achieves a superior Pareto frontier between fidelity and diversity, outperforming traditional methods like temperature scaling and discrete guidance baselines in reasoning-intensive tasks such as code generation and mathematical problem-solving.