ImmersiveTTS: AI model generates speech that blends naturally with any environment

ImmersiveTTS tackles the challenge of jointly generating speech and environmental audio—a notoriously difficult task due to the mismatched acoustic patterns and temporal dynamics between voiced speech and background sounds. The model, from researchers at Korea University (Seong-Whan Lee's group), builds on a multimodal diffusion transformer architecture. It first extracts transcript-aligned speech latents from the input text, then fuses these with text-conditioned environmental context via a joint attention mechanism. This cross-modal interaction allows the model to place speech naturally within a scene—for example, having someone speak in a rainstorm without the voice sounding artificial or detached.

To ensure semantic consistency between speech content and the audio environment, the team introduces a domain-specific representation alignment objective. This leverages complementary self-supervised representations from separate speech and audio encoders, aligning them at a latent level. Experimental results, measured across objective metrics (e.g., MOS, WER) and human listening tests, show that ImmersiveTTS outperforms existing text-to-speech and audio generation methods in naturalness, intelligibility, and overall fidelity. The code is available on GitHub, and the paper has been accepted to ACL 2026. This work opens the door for more immersive spoken dialogue in VR, assistive technology, and media production.