iFLYTEK's Embodied-Omni unifies vision, language, and action in one model

iFLYTEK's new Embodied-Omni model tackles a core challenge in embodied AI: unifying perception, world modeling, and action generation. Most existing systems chain separate models for visual-language reasoning, future video prediction, and control, creating interface bottlenecks and compounding errors. Embodied-Omni instead uses a single Omni framework with shared multimodal self-attention, allowing its vision-language model (VLM), video generation model (VGM), and action generation model (AGM) to communicate directly. The design mimics brain-cerebellum collaboration: the VLM and VGM act as the high-level brain for instruction understanding, task planning, progress tracking, and forecasting future visual states, while the AGM serves as the low-level cerebellum that converts planned subgoals and shared context into executable action chunks.

To train this system, iFLYTEK curated a comprehensive dataset combining action-annotated and action-free embodied videos from human demonstrations and robot interactions, plus embodied reasoning, perception, and general-purpose image-text data. They employed a four-stage training strategy: first pre-training the VLM, then the VGM, then the AGM, and finally joint fine-tuning of the entire model. This approach ensures each modality builds on shared representations while maintaining specialization. The result is a general-purpose embodied agent capable of understanding multimodal instructions, anticipating environment evolution, and producing precise, long-horizon control actions—all without cascading errors.