StyleVAR: Controllable Image Style Transfer via Visual Autoregressive Modeling

StyleVAR, introduced by Liqi Jing and colleagues, reframes image style transfer as a conditional discrete sequence modeling problem within the Visual Autoregressive Modeling (VAR) framework. Images are decomposed into multi-scale representations and tokenized into discrete codes using a VQ-VAE. A transformer then autoregressively models the distribution of target tokens, conditioned on both style and content tokens. To effectively blend these influences, the team developed a blended cross-attention mechanism where the evolving target representation attends to its own history, while style and content features act as queries to emphasize relevant aspects. A scale-dependent blending coefficient controls the relative influence of style and content at each stage, ensuring the synthesized representation aligns with content structure and style texture without breaking autoregressive continuity.

The training process occurred in two stages: first, supervised fine-tuning on a large dataset of content-style-target triplets; second, reinforcement fine-tuning using Group Relative Policy Optimization (GRPO) with a DreamSim-based perceptual reward and per-action normalization weighting. Across three benchmarks covering in-, near-, and out-of-distribution scenarios, StyleVAR consistently outperformed an AdaIN baseline on metrics including Style Loss, Content Loss, LPIPS, SSIM, DreamSim, and CLIP similarity. The GRPO stage yielded further gains, particularly on reward-aligned perceptual metrics. Qualitatively, StyleVAR excels at transferring texture while preserving semantic structure in landscapes and architectural scenes, though it struggles with internet images and human faces, highlighting areas for future improvement.