Learning Illumination Control in Diffusion Models

Researchers from the University of Maryland and IIT Delhi have developed a fully open-source pipeline for controlling illumination in images using diffusion models. The team—Nishit Anand, Manan Suri, Christopher Metzler, Dinesh Manocha, and Ramani Duraiswami—built a data engine that transforms well-lit images into supervised training triplets. Each triplet consists of a poorly-illuminated input image, a natural language lighting instruction (e.g., "make the room brighter with warm tones"), and a well-illuminated output image.

When fine-tuned on this data, their diffusion model achieves significant improvements over baseline models like Stable Diffusion 1.5, SDXL, and FLUX.1-dev in perceptual similarity, structural similarity, and identity preservation. The work was accepted to ICLR 2026's ReALM-GEN Workshop on Diffusion Models. By releasing all code, data, and model weights publicly, the team provides a reproducible solution that eliminates the need for heavy control inputs like depth maps, making advanced illumination control accessible to the open-source community.