Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

A team of researchers, including Ahmed Bourouis, Savas Ozkan, and Andrea Maracani, has introduced a groundbreaking AI framework that tackles the novel problem of generating consistent 3D scenes from a single, geometrically impoverished freehand sketch. Unlike existing methods that require photographs, text, or multiple input views, their system directly interprets abstract, distorted 2D sketches to produce a full multi-view scene in a single denoising process, eliminating the need for iterative refinement or costly per-scene optimization.

To overcome the lack of training data and the challenge of deriving 3D geometry from distorted 2D strokes, the team made three key contributions. First, they created a curated dataset of approximately 9,000 sketch-to-multiview samples using an automated generation and filtering pipeline. Second, they developed Parallel Camera-Aware Attention Adapters (CA3) to inject crucial geometric inductive biases into a video transformer backbone. Third, they introduced a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions to enforce cross-view consistency.

The results are significant. The framework substantially outperforms prior two-stage baselines, improving the realism of generated scenes by over 60% as measured by the Fréchet Inception Distance (FID) metric. It also boosts geometric consistency (Corr-Acc) by 23% while providing up to a 3.7x inference speedup. This represents a major leap in making 3D content creation accessible, allowing users to generate complex, consistent 3D scenes from the simplest of 2D inputs.