SupraLabs' New Model Has No Vision Encoder — Yet It Unifies Text, Image, Video in One Token Stream

SupraLabs introduced Supra-A2A-Nano-Exp, a ~30M parameter autoregressive Transformer that rethinks multimodality by treating everything as tokens in a unified stream. Unlike traditional multimodal models that rely on separate vision encoders, diffusion models, or cross-attention modules, this model tokenizes text, image patches, and video frames into a shared vocabulary of 50,520 tokens. Text uses GPT-2 BPE (50,264 tokens) while images are encoded via a 3-layer VQ-VAE with 256 codes, creating an 8x8 grid. The model simply predicts the next token, whether it's a word, a visual code, or a frame marker. It supports text generation, chat, image reconstruction, and text-to-image generation through a single inference pipeline.

This radical simplification—modality-agnostic next-token prediction—makes the architecture elegant but carries significant limitations. With only 384 tokens of context, ~30M parameters, and no RLHF or instruction tuning, outputs are low-resolution and abstract. The VQ-VAE output activation is assumed sigmoid (0–1), but tanh is also possible. Attention heads are inferred as 4 but may be incorrect. Despite these caveats, the model demonstrates that separate systems for vision and language may be unnecessary. It's an experimental research prototype aimed at exploring the idea that modality boundaries disappear when all data is represented as tokens in a single Transformer.