MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

Researchers Bharath Krishnamurthy and Ajita Rattani have introduced MMFace-DiT, a unified dual-stream diffusion transformer engineered specifically for synergistic multimodal face synthesis. The core innovation lies in its dual-stream transformer block architecture that processes spatial tokens (from masks or sketches) and semantic tokens (from text) in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents any single modality from dominating the generation process, ensuring strong adherence to both textual descriptions and structural priors. The model achieves a remarkable 40% improvement in visual fidelity and prompt alignment compared to six existing state-of-the-art multimodal face generation models.

Unlike previous approaches that typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks, MMFace-DiT offers a cohesive end-to-end solution. The novel Modality Embedder enables the single model to dynamically adapt to varying spatial conditions without requiring retraining, addressing the limitations of ad hoc designs that often fail under conflicting modalities or mismatched latent spaces. The research, accepted to CVPR 2026, establishes a flexible new paradigm for controllable generative modeling where users can generate faces with unprecedented spatial-semantic consistency, precisely following both high-level semantic intent and low-level structural layout simultaneously.