DiffusionGemma's latent reasoning stays interpretable, rare vector cases found

Google DeepMind's DiffusionGemma (DG) generates text through reverse diffusion, passing both tokens and probability-distribution vectors between steps. That vector channel creates opaque serial depth—a priori, the model could hide substantial computation in illegible distributions. In a new AI Alignment Forum analysis, Jan Bauer and Neel Nanda test whether DG actually uses this latent channel in ways that break monitorability. They find that projecting the distribution to its single top token retains strong performance, meaning most quality degradation is just a sampling artifact, not evidence of hidden reasoning. But they also identify rare cases where the distribution vector is genuinely load-bearing—computing things that top-1 projection cannot preserve.

Crucially, even those adversarial cases remain interpretable: the vector encodes superposition, a well-understood phenomenon where multiple features share representational dimensions. The authors also validated that interpretability tools carry over to DG—linear probes, steering vectors, and J-lens (a technique for inspecting intermediate representations) all retain their effectiveness. This is a positive update for diffusion models derived from text-pretrained LLMs like Gemma, an efficient training approach likely to see wider deployment. It does not guarantee that more exotic diffusion paradigms will stay transparent, but it narrows the gap between autoregressive and diffusion-based interpretability. For practitioners, this suggests DG's latent reasoning can be audited without sacrificing the architectural benefits of parallel, bidirectional generation.