Pareto LoRA boosts multimodal model image quality by 44.9%

Unified multimodal models (UMMs) that handle both text and image generation often suffer from a severe modality imbalance during parameter-efficient fine-tuning (PEFT) like LoRA. The authors show that language gradients dominate the optimization process, causing vision performance to degrade substantially more than text when compared to unimodal counterparts. In some layers, modality-specific gradients differ by orders of magnitude, leading to poor image generation quality even as text remains strong.

To solve this, the team reformulates multimodal instruction tuning as a bi-objective optimization problem and proposes Pareto LoRA. This strategy integrates gradients from text and image objectives by adjusting both direction and magnitude to find a Pareto-optimal balance. Tested on the CoMM benchmark with the Emu2 architecture, Pareto LoRA achieved up to 44.9% gains in perceptual image quality (e.g., FID and CLIP scores) while keeping text performance on par with vanilla LoRA. This approach offers a principled way to boost multimodal generation without sacrificing efficiency.