Omni-Flow unifies multimodal AI inference with distributed KV cache sharing
New framework cuts GPU memory waste by sharing KV caches across models and nodes.
Omni-Flow introduces a three-layer abstraction to unify multimodal inference workflows. The Control Flow layer lets developers define pipelines as a DAG using a Python DSL, supporting static graphs, dynamic routing, and built-in load balancing. The Data Flow layer tackles the massive intermediate data movement problem with a distributed KV cache abstraction that spans GPU, CPU, and SSD in a paged hierarchy, enabling zero-copy, low-latency sharing across nodes. The Compute Flow layer provides complex multimodal prefix matching for KV reuse across conversations and takes over sampling logic via a unified SGLang interface, allowing diffusion models to reuse LLM forward passes.
The framework addresses a gap in current systems that deploy LLMs and diffusion models separately, causing scattered orchestration and tight coupling between data paths and specific models. By abstracting control, data, and compute into reusable layers, Omni-Flow enables heterogeneous models to cooperate under a single programming model. It demonstrates support for two demanding scenarios: omni-modal dialogue (LongCat-Next) and complex image generation (HunyuanImage-3). The result is lower memory footprint, faster multi-turn inference, and easier integration of new model types into production pipelines.
- Three-layer abstraction (Control, Data, Compute Flow) separates orchestration, data transfer, and compute logic for multimodal pipelines.
- Distributed KV cache with three-tier paged storage (GPU/CPU/SSD) enables zero-copy cross-role sharing, eliminating redundant GPU memory.
- Supports complex multimodal prefix matching for KV reuse across multi-turn dialogues, demonstrated on LongCat-Next and HunyuanImage-3.
Why It Matters
Makes multimodal AI systems faster, cheaper, and easier to build by unifying LLMs and diffusion models under one framework.