WidgetGen framework beats pipelines for screenshot-to-UI-code conversion
WidgetGen cuts pipeline complexity while improving visual fidelity across 6 models and 1,000 widgets.
Screenshot-to-code systems often struggle to balance flexibility with controllability. Direct multimodal generation hallucinates visual details, while structured pipelines reduce errors through component-wise decomposition and predefined templates—but at the cost of restrictive output schemas and extra generative orchestration. In a paper accepted to the ECCV 2026 MUCG Workshop, researchers propose WidgetGen, a lightweight alternative that selectively grounds the model in observable evidence rather than forcing it through a fixed UI representation.
WidgetGen extracts visible text and color cues from a widget, performs high-level layout and optional chart reasoning, and then directly emits executable JavaScript XML (JSX). The approach avoids component-wise generation while maintaining output freedom. Evaluated on six multimodal models against 1,000 held-out widgets, WidgetGen surpassed both direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Furthermore, the team used the reconstruction-derived image-code pairs to supervised fine-tune six Qwen-family open-weight models, improving every reported metric. These results position WidgetGen as a strong lightweight baseline and demonstrate that selective tool grounding can rival extensive representation engineering.
- WidgetGen generates executable JSX directly from UI screenshots, skipping intermediate component templates.
- Outperforms direct prompting and Widget2Code across 6 multimodal models on 1,000 held-out widgets.
- Reconstruction pairs boost 6 Qwen open-weight models across all reported metrics via supervised fine-tuning.
Why It Matters
Faster, higher-fidelity UI code generation means developers can prototype from mockups with less manual cleanup and fewer hallucinations.