NVIDIA's TRON merges ray tracing and neural rendering for editable 3D scenes

NVIDIA Research has unveiled TRON, a novel rendering framework that bridges physically based rendering (PBR) and neural rendering for high-fidelity 3D scene editing. Traditional Gaussian-based PBR relighting suffers from imperfections in reconstructed geometry and inaccurate material estimates, while neural rendering often lacks explicit scene representations needed for fine-grained interactive control. TRON solves this by using intrinsic decomposition priors from a learned inverse rendering model to regularize material properties of a 3D Gaussian field. It then repurposes a ray tracer to provide radiometric guidance—not final pixels—as a structured 3D scaffold for a lightweight neural renderer. This combination delivers the speed and controllability of explicit 3D knowledge with the photorealistic output of neural synthesis.

To support real-world use, TRON employs a multi-stage training strategy with large-scale pretraining and targeted fine-tuning on a newly constructed dataset of 2.1 million rendered frames from synthetic and real-world 3D reconstructions. The result outperforms existing Gaussian-based relighting methods in realism and prior neural renderers in both editability and speed. TRON is the first method to enable practical interactive applications in captured 3D environments, allowing users to realistically adjust lighting, insert dynamic objects, and edit materials in real time—a significant leap for industries like film, gaming, VR/AR, and digital twins.