TDMM-LM: Bridging Facial Understanding and Animation via Language Models

A team of researchers from institutions including the University of Michigan and the University of Rochester has published TDMM-LM, a groundbreaking framework that bridges facial understanding and animation using language models. The central challenge they address is the scarcity of high-quality, text-paired facial motion data, which has hindered progress compared to full-body animation. Their solution was to leverage existing foundation generative models to synthesize a massive, balanced corpus of facial behavior. By designing a comprehensive prompt suite covering emotions and head motions, they generated approximately 80 hours of facial videos, fitted them with per-frame 3D facial parameters, and created a large-scale dataset of (text prompt, 3D parameter) pairs for training.

Building on this novel dataset, the team probed the capabilities of language models for bidirectional facial motion tasks. The first task, Motion2Language, requires the model to analyze a sequence of 3D facial parameters and produce a natural-language description capturing the content, style, and dynamics of the motion. The second, Language2Motion, involves the model taking a text prompt and synthesizing the corresponding sequence of 3D facial parameters, which are represented as quantized motion tokens for use in downstream animation systems. Extensive experiments demonstrate that language models trained in this setting can both interpret and synthesize facial motion with strong generalization.

This work is significant because it is the first to cast 3D facial-parameter modeling entirely as a language problem. By representing facial motion as a sequence of discrete tokens—similar to how words are tokenized in text—TDMM-LM establishes a unified pipeline for text-conditioned facial animation and motion understanding. This approach could dramatically simplify the creation of expressive digital avatars for film, gaming, and virtual communication, moving beyond pre-canned animations to dynamic, prompt-driven facial performances.