AVTok: A unified 1D tokenizer for joint audio-video generation

Current audio-video generation models typically use separate tokenizers and generators for each modality (dual-branch design), which ignores the representation gap and consumes heavy compute. AVTok, presented by Pham et al. at ECCV 2026, breaks this pattern with a single tokenizer that maps both audio and video into a compact 1D latent space using a unified codebook. Its architecture features a dual-stream transformer with shared encoder-decoder and modal-specific learnable queries, allowing efficient encoding of heterogeneous data. To handle the imbalance between audio and video information, the authors introduce a hierarchical training strategy that progressively builds reconstruction capabilities for each modality.

Extensive experiments show AVTok achieves strong performance in reconstruction and integrates seamlessly into downstream pipelines: audio-to-video, video-to-audio, and class-conditional joint generation. By unifying tokenization, AVTok reduces computational overhead and opens a path toward building large multimodal models that process audio and video holistically. This work marks a significant step beyond separate-modality approaches, especially for synchronized, semantically aligned content creation.