Why few-step text generation fails: decoder sharpness causes token flipping
Continuous text latents collapse to gibberish while image latents converge – here's the geometric cause.
A new theoretical paper by Zhongyao Wang tackles a puzzling asymmetry in generative AI: why does deterministic few-step generation succeed on continuous image latents but collapse to incoherent text when applied to continuous text latents? The answer, according to the paper, is geometric rather than a training or scaling issue. The key finding is that text decoders exhibit extreme "readout sharpness" – a property measured by a new diagnostic called DABI (Decoder Alignment-Based sharpness Index). While image decoders have DABI ≈ 1, four independently built continuous-text decoders show DABI ranging from 5×10² to over 10⁵, meaning they amplify small perturbations aligned with decision boundaries far beyond isotropic noise.
The paper proves (Theorem 3) that in the overlapping regime of real text autoencoders, the posterior-mean terminal step flips tokens at a rate proportional to the latent mass in an O(s(t)) tube around decision boundaries. This leads to two identified escape mechanisms: categorical commitment (autoregressive decoders succeed despite even sharper readouts) and stochastic re-injection (deterministic ODE at K=4 gives perplexity 294 vs SDE 50 on the same model). The work also establishes matching sharp transport laws and a dimension phase diagram, showing deterministic stiffness must grow as Θ(√log M) once latent dimension is Ω(log M). The result is an accuracy-depth-stiffness tradeoff that is irreducible within the deterministic-continuous class.
- Four continuous-text decoders measured DABI from 5×10² to >10⁵, while image decoders had DABI ≈ 1, confirming extreme decoder sharpness is the core issue.
- Stochastic re-injection dramatically outperforms deterministic methods: SDE achieves perplexity 50 vs ODE's 294 on the same model at K=4 steps.
- The paper proves a fundamental tradeoff: deterministic-continuous few-step methods incur irreducible cost from an accuracy-depth-stiffness tradeoff, forcing either autoregressive or stochastic approaches.
Why It Matters
Explains why fast text generation breaks and points to stochastic or autoregressive methods as necessary workarounds for practitioners.