From "What" to "How": Constrained Reasoning for Autoregressive Image Generation

A research team led by Ruxue Yan has introduced CoR-Painter, a groundbreaking framework that fundamentally shifts how AI models approach image generation from text. The core innovation addresses a critical weakness in current autoregressive models like DALL-E 3 or Stable Diffusion: they excel at interpreting 'What' to draw by rewriting prompts but fail at reasoning 'How' to structurally compose the scene. This leads to persistent issues like unrealistic object overlaps and spatial ambiguity. CoR-Painter pioneers a 'How-to-What' paradigm, where the model first deduces a set of explicit visual constraints—governing spatial relationships, key attributes, and compositional rules—before generating any detailed description or pixels.

The technical engine of CoR-Painter is its Constrained Reasoning module and a Dual-Objective GRPO (likely a variant of Group Relative Policy Optimization) training strategy. The system explicitly reasons about layout, ensuring objects are placed coherently before detailing their appearance. This structured approach resulted in significant performance gains, with a +5.41% improvement on the challenging T2I-CompBench benchmark, which tests compositional understanding. The method demonstrates state-of-the-art results on GenEval and WISE benchmarks as well. This research, detailed in the arXiv paper 'From "What" to "How": Constrained Reasoning for Autoregressive Image Generation,' points toward a future where AI image generators are not just prompt-followers but capable of spatial planning and coherent scene construction.